docs: M1 Ultra fmt=2 Metal perf report + benchmarks row + fmt4→fmt2 converter - #836
Merged
Conversation
…ans, scripts Goal a: reproduce the issue JustVugg#103 M5 Max result (2.06-2.24 tok/s) on M1 Ultra using fmt=2 (per-row int4) weights, which the Metal backend fully supports. - docs/issue_103.md: the reproduction target (full original issue text) - docs/metal_fmt2.md: fmt=4 -> fmt=2 conversion + benchmark plan (Configs A-E) - docs/ultra_benchmark_plan.md: overall M1 Ultra plan (both tracks) - docs/metal_dispatch_gap.md: why the fmt=2 track exists (fmt=4 Metal gap) - docs/metal_issues.txt: master index of Metal issues/PRs - Issue summaries: benchmark datapoints (JustVugg#47 JustVugg#87 JustVugg#107 JustVugg#180), tuning A/Bs (JustVugg#387), cache-slowdown diagnosis (JustVugg#379), OMP spin-wait (JustVugg#707), prefill GEMM token-exactness caveat (JustVugg#622) - PR summaries: Metal backend (JustVugg#72), M5 Max report + methodology (JustVugg#116), platform-aware cache defaults (JustVugg#386), tuned Apple defaults (JustVugg#750), shared-experts GPU technique (JustVugg#757), GPU prefill attention option (JustVugg#763) - c/convert_fmt4_to_fmt2.sh: the fmt=4 -> fmt=2 converter invocation - c/x.sh, c/venv.sh: benchmark runner + python env helper
… converter Safety checkpoint before restructuring. Contains the full ephemeral workspace: v1.4.0 run logs + SUMMARY, pre-rebase v1.2.0 logs (fmt2.old), benchmark harness, usage snapshot, context docs, and the staged fmt4->fmt2 converter. None of the docs/fmt2* content is intended for the upstream PR; it will be untracked and excluded in a later commit.
…rter PR content (the whole intended upstream diff): - docs/METAL-M1ULTRA-FMT2-REPORT.md: fmt=2 on M1 Ultra, best 1.50 tok/s @ --ram 125 vs M5 Max 2.24 (-33%) with near-equal GPU cores (48 vs 40). Disk wait is 55-60% of the serial decode wall and the drive runs at ~93% of its 6.89 GB/s iobench F_NOCACHE ceiling in-decode: the SSD, not the GPU, sets the speed. OMP spin trap absent; PIPE is the only lever (+6.9%); PIPE_WORKERS=8 sweet spot; MTP strict loss at 128 GB. - docs/benchmarks.md: community row (same shape as the M5 Max JustVugg#103 row) + one Takeaways sentence (M1 Ultra <-> M5 Max as the GPU-core-count control). - c/tools/convert_fmt4_to_fmt2.py: the fmt=4 (g64) -> fmt=2 (per-row) re-quant converter used to build the benchmark container (--selftest passes); the only in-tree path to an fmt=2 container while Metal fmt=4 dispatch is open (JustVugg#585/JustVugg#587). One-line c/tools/README.md entry. The ephemeral benchmark workspace (logs, SUMMARY, harness, usage snapshot, plan/context docs) stays on disk under docs/fmt2/ but is untracked and locally excluded (.git/info/exclude) — not part of the PR.
JustVugg
added
benchmark
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
M1 Ultra counterpart to the M5 Max Metal report (#116, data from #103): same methodology, same prompt, same frozen expert history, on fmt=2 (per-row int4) weights — the format the Metal backend fully supports while fmt=4 dispatch is open (#585/#587).
Headline: best 1.50 tok/s (
--ram 125, tuned) vs the M5 Max's 2.24 — −33% with near-equal GPU core counts (48 vs 40). Disk wait is 55–60% of the fully serial decode wall, and the drive runs at ~93% of its 6.89 GB/s iobench (F_NOCACHE) ceiling in-decode: on this workload the SSD, not the GPU, sets the speed.Secondary findings: the M5 Max OMP active-spin trap does not reproduce on M1 Ultra;
PIPEis the only tuning lever (+6.9%),PIPE_WORKERS=8the sweet spot; MTP is a strict loss at 128 GB (memory knee → swap thrash); v1.4.0's fast-SSD cap=1 default costs −33% here (--cap 33restores it); v1.2.0→v1.4.0 rebase is performance-neutral.Files (4)
docs/METAL-M1ULTRA-FMT2-REPORT.md— full report (setup, results, analysis, rebase check, recommendations, caveats)docs/benchmarks.md— one community-benchmarks row + one Takeaways sentencec/tools/convert_fmt4_to_fmt2.py— the fmt=4 (g64) → fmt=2 (per-row) re-quant converter used to build the benchmark container (--selftestpasses). The only in-tree path to an fmt=2 container while Metal fmt=4 is open — happy to drop it from the PR if you'd rather not carry a GLM-5.2-specific tool.c/tools/README.md— one-line entry for the converterNotes for review
c024a46(v1.4.0); single run per config (±0.05 tok/s noise) — repeats for confidence intervals are listed as the follow-up.