Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
67 commits
Select commit Hold shift + click to select a range
f3b5c5b
Add Kimi K3 reference model
JavaZeroo Jul 30, 2026
8de1124
Remove development-only Kimi K3 diagnostics
JavaZeroo Jul 30, 2026
bf7c6d5
Add FSDP2 support for Kimi K3
JavaZeroo Jul 30, 2026
31fd2d7
Scale Kimi K3 debug model
JavaZeroo Jul 30, 2026
51b7007
Use out-of-place operations in Kimi K3 eager paths
JavaZeroo Jul 30, 2026
a626834
Keep unused Kimi experts in the FSDP autograd graph
JavaZeroo Jul 30, 2026
bedf145
Add Kimi K3 numerical and mixed-modality FSDP coverage
JavaZeroo Jul 31, 2026
6da10ec
Reuse shared components in Kimi K3 and run KDA on the FLA kernel
JavaZeroo Jul 31, 2026
e87a87a
Guard Kimi K3 optional deps and cover the residual pass-through path
JavaZeroo Aug 1, 2026
6d2d190
Drop the KDA head-dim guard
JavaZeroo Aug 1, 2026
f4b3808
Close the Kimi K3 backbone on global attention
JavaZeroo Aug 4, 2026
25577e4
Note the context-parallel route in add_zero_valued_dependency
JavaZeroo Aug 4, 2026
4f7f668
reuse common FeedForward
JavaZeroo Aug 9, 2026
611dd73
dp_replicate needs no specific handling, removed from unsupported_par…
JavaZeroo Aug 9, 2026
d05952e
Refactor KimiGroupedExperts to use gmm
JavaZeroo Aug 10, 2026
6fe3694
refactor KimiMLAAttention to Inheritance titian BaseAttention
JavaZeroo Aug 10, 2026
1fc70f6
remove add_zero_valued_dependency, this should be address with other …
JavaZeroo Aug 10, 2026
522958f
remove redundandency test case
JavaZeroo Aug 11, 2026
51ad154
connect FLA operator, replace RMSNorm
the-fall-moon Aug 11, 2026
bc0e271
revert fla operator
the-fall-moon Aug 11, 2026
c5b9330
remove test_kimi_k3_hf_parity.py
JavaZeroo Aug 11, 2026
762f4f4
connect FLA operator, replace RMSNorm
the-fall-moon Aug 11, 2026
4f972ba
resotre to eager conv and RMSNormGated
JavaZeroo Aug 11, 2026
dc009a9
remove KimiExactGELU, replaced with nn.GELU
JavaZeroo Aug 11, 2026
d37a4a5
Add Kimi K3 numerical validation
JavaZeroo Aug 13, 2026
2509f18
remove parallelism config
JavaZeroo Aug 14, 2026
89d93a9
change full_attention_layers to starting with 0
JavaZeroo Aug 14, 2026
ae006e4
support flex attention
JavaZeroo Aug 15, 2026
fb7865a
refactor KimiK3StateDictAdapter
JavaZeroo Aug 15, 2026
9f02c89
refactor KimiGroupedExperts to use self.grouped_mm
JavaZeroo Aug 15, 2026
588c62e
use snapshot_download to pin the version
JavaZeroo Aug 16, 2026
edccb38
refator test case
JavaZeroo Aug 16, 2026
02960f7
fix pyrefly
JavaZeroo Aug 16, 2026
2e7603a
support full kimi k3 and refactor readme
JavaZeroo Aug 17, 2026
22cfd54
fix some NIT and some bug
JavaZeroo Aug 17, 2026
f7feaaa
reuse VisionAttention
JavaZeroo Aug 17, 2026
af8b872
simplify test_kimi_k3.py
JavaZeroo Aug 17, 2026
f8523a2
reuse common MoonViT helpers, VisionTransformerBlock, and MoE
JavaZeroo Aug 18, 2026
71c9804
address review feedback on the debug model, tests, and README
JavaZeroo Aug 18, 2026
8d71077
register kimi_k3 in the CLI option freeze
JavaZeroo Aug 18, 2026
e6e6e64
fix unused change
JavaZeroo Aug 19, 2026
905b366
reuse ComplexRoPE
JavaZeroo Aug 19, 2026
0f45ff8
scale up k3 model
JavaZeroo Aug 19, 2026
6273274
remove nit
JavaZeroo Aug 19, 2026
d56c69e
remove unsed head_dim in KimiDeltaAttention
JavaZeroo Aug 19, 2026
5e006ba
remove unsed riase Error
JavaZeroo Aug 19, 2026
99c80c0
remove some redundancy assert
JavaZeroo Aug 19, 2026
f925dad
fix activation checkpointing
JavaZeroo Aug 19, 2026
950f547
Add TODO for _apply_attention_residual
JavaZeroo Aug 19, 2026
6689074
split k3 model.py to moe.py lda.py
JavaZeroo Aug 20, 2026
252ae22
remove two unsed attn res from hf, add it in state_dict_adapter
JavaZeroo Aug 21, 2026
2ef1981
numerical_tests_kimi_k3.py support different attn_backend and bf16
JavaZeroo Aug 22, 2026
41447a4
Adapt Kimi K3 to the folded token layout
JavaZeroo Aug 22, 2026
4149422
update sample packing checking
JavaZeroo Aug 22, 2026
9f8bcf0
add TODO
JavaZeroo Aug 22, 2026
44d0e83
chang fla kda from fp32 to bf16
JavaZeroo Aug 22, 2026
dee45e3
fix test_update_from_config_propagates_moe_force_load_balance when re…
JavaZeroo Aug 22, 2026
84e9f7b
rename probs_TN to probs_T1N
JavaZeroo Aug 23, 2026
157440a
removed dtype args, only support bf16 now
JavaZeroo Aug 23, 2026
075d54c
Clarify the README that the numerical results are based on a reduced …
JavaZeroo Aug 23, 2026
99c2c68
remove test_update_from_config_propagates_moe_force_load_balance
JavaZeroo Aug 23, 2026
0999206
remove TestKimiK3FSDP, and add this in to github ci
JavaZeroo Aug 23, 2026
32abaa0
reuse some function from k2.7
JavaZeroo Aug 23, 2026
ded1bf1
share the MoonViT tower between k2.7 and k3
JavaZeroo Aug 23, 2026
b3751c9
Update .github/workflows/integration_test_8gpu_features.yaml
JavaZeroo Aug 24, 2026
2dd9390
fix cicd
JavaZeroo Aug 24, 2026
88cb800
Update scripts/checkpoint_conversion/numerical_tests_kimi_k3.py
shuhuayu Aug 24, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion .github/workflows/integration_test_8gpu_features.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ jobs:
TORCH_SPEC="torch==${{ matrix.torch-version }}"
fi
python -m pip install --force-reinstall --pre \
"${TORCH_SPEC}" --index-url ${{ matrix.index-url }}
"${TORCH_SPEC}" torchvision --index-url ${{ matrix.index-url }}

# The torchcomms feature tests are currently disabled, so do not install
# torchcomms in the main feature job. Its wheel pins torch exactly and
Expand All @@ -93,6 +93,12 @@ jobs:
python -m pytest tests/unit_tests/flex_shard/test_dist_muon.py \
--durations=20 -vv

if [[ "${{ matrix.gpu-arch-type }}" == "cuda" ]]; then
CUDA_VISIBLE_DEVICES=0 python -m pytest \
tests/unit_tests/test_kimi_k3.py::TestKimiK3::test_fla_kda_kernel_matches_recurrent_reference \
-vv
fi

sudo mkdir -p "$RUNNER_TEMP/artifacts-to-be-uploaded"
sudo chown -R $(id -u):$(id -g) "$RUNNER_TEMP/artifacts-to-be-uploaded"

Expand Down
Loading