omni: INT4 per-block zero-point GEMM + calling adapter (torchao asymmetric INT4) - #629
Open
JWLHS wants to merge 2 commits into
Open
omni: INT4 per-block zero-point GEMM + calling adapter (torchao asymmetric INT4)#629JWLHS wants to merge 2 commits into
JWLHS wants to merge 2 commits into
Conversation
…tric INT4) onednn_int4_gemm_preconverted gains an optional zp_u8 [G, N]. When provided it applies per-block zero points inside oneDNN, w = (q - zp) * scale — native support for INT4 models quantized on the torchao backend (Int4PlainInt32Tensor: int32 qdata byte view + per-block zp/scale, asymmetric). The per-block zero point centers biased weight distributions per block, lowering quantization error vs symmetric INT4.
- adapters/int4_gemm.py: int4_gemm wrapper over onednn_int4_gemm_preconverted with optional zp_u8 + capability check - config flag OMNIXPU_INT4_GEMM + patch component registration
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
目的
为 llm-scaler 的 omni 组件增加 torchao 后端量化的非对称 INT4 模型 支持：
zp_u8 [G, N] 参数，提供时在 oneDNN 内应用 per-block zero point，
w = (q - zp) * scale —— 原生支持 Int4PlainInt32Tensor（int32 qdata 字节
视图 + per-block zp/scale，非对称）。零转换、零 Python 修正项。
（int4_gemm，透传 zp_u8）+ 能力探测 + config/组件注册，与 kernel 改动配套。
非对称 per-block zero point 能把偏置的权重分布逐块归中，相比对称 INT4
量化误差更小（编辑/姿态迁移类能力保留更好）。
改动
配套插件（用于体验与测试）
统一加载器 JWLHS/int4-omnixpu：wa4 + torchao
INT4 统一加载，探测本算子存在即原生调用（无则安全回退 torchao/python），开箱即用。