Skip to content

fix(ftw): preserve auxiliary expert banks - #317

Open
ErikBPF wants to merge 1 commit into
FlashML-org:mainfrom
ErikBPF:fix/ftw-auxiliary-banks
Open

fix(ftw): preserve auxiliary expert banks#317
ErikBPF wants to merge 1 commit into
FlashML-org:mainfrom
ErikBPF:fix/ftw-auxiliary-banks

Conversation

@ErikBPF

@ErikBPF ErikBPF commented Sep 1, 2026

Copy link
Copy Markdown

Summary

Preserve expert banks that use a different quant format or row geometry from the primary bank set during FTW conversion and load. Without this, converting a heterogeneous expert layout silently writes only the primary banks.

  • add an optional auxiliary-bank contract to ExpertBanks
  • write auxiliary banks as a nested FTW checkpoint using the existing writer
  • record the nested checkpoint and model-layer mapping in parent metadata
  • validate that checkpoint and mapping metadata appear together
  • recursively load the nested banks and restore their quant format, sources, and layer IDs
  • leave existing FTW checkpoints unchanged

Provider-specific Qwen loading is intentionally separate; this PR contains only the generic FTW preservation mechanism.

Tests

  • writer test verifies per-layer nested entries and metadata
  • loader test verifies quant format, layer mapping, and exact tensor round-trip
  • Python compile, Ruff on the new test, and git diff --check pass

Full-model evidence

Matched Orion ROCm runs used 8,192-token context, 1,024 output tokens, concurrency 4, 3 rounds, and 12 requests per format:

Format Aggregate decode Aggregate E2E Median TTFT
GGUF 111.9149 tok/s 107.2566 tok/s 1.7146 s
FTW 113.6572 tok/s 108.8996 tok/s 1.6936 s

All 24 requests finished by length. Every GGUF and FTW output matched SHA-256 1ec7793ebf17b06a81148a198b7fbb5e5a68018c75982dd123cb1cf0fc2b8860. Both formats used 2,073 expert slots, 8,219 KV pages, and retained 7.71 GiB free after initialization.

The primary claim is exact conversion/load preservation. The observed +1.56% decode difference is secondary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant