feat(kvcache): add reliable quantized KV cache - #309

Open

plsgivemeachane wants to merge 5 commits into

FlashML-org:mainfrom

plsgivemeachane:feat/reliable-quantized-kv-cache

Commits on Aug 31, 2026

feat(kvcache): sub-byte Q4_0 KV cache quantization

fangyuan-3149
authored and
plsgivemeachane
committed
c086b28
fix(kvcache): wire --kv-cache-dtype through the engine

fangyuan-3149
authored and
plsgivemeachane
committed
99bd90c
fix(tests): correct sub-byte test expectations and dtype mismatches

fangyuan-3149
authored and
plsgivemeachane
committed
09c9d94
fix(kvcache): correct quantized KV attention data path
plsgivemeachane
committed
aad9a16
fix(attention): keep sink normalizer in fp32
plsgivemeachane
committed
716c1a2