From b926fae9c5bf8485380c9829be3fe77404bdd8b4 Mon Sep 17 00:00:00 2001 From: Pablo Gonzalez Date: Wed, 29 Jul 2026 19:57:56 -0500 Subject: [PATCH] Add MLPerf Inference v6.1 benchmarks table --- README.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/README.md b/README.md index bdf3c3a44c..81fb646d6e 100644 --- a/README.md +++ b/README.md @@ -17,6 +17,35 @@ Please see the [MLPerf Inference benchmark paper](https://arxiv.org/abs/1911.025 Please see [here](https://docs.mlcommons.org/inference/benchmarks/) for the MLPerf inference documentation website which includes automated commands to run MLPerf inference benchmarks using different implementations. +## MLPerf Inference v6.1 (submission deadline July 31 2026) + +For submissions, please use the master branch and any commit since the [v6.1 seed release](https://github.com/mlcommons/inference/commit/b0662794362bc2c22c33c3e4a0c66ef83411ecf0) although it is best to use the latest commit in the [master branch](https://github.com/mlcommons/inference). + +For power submissions please use [SPEC PTD 1.11.1](https://github.com/mlcommons/power) (needs special access) and any commit of the power-dev repository after the [code-freeze](https://github.com/mlcommons/power-dev/commit/c4b3ad8202fbd8ac28d77149e5e7aeadb725bbf2) + + +| model | reference app | framework | dataset | category +| ---- | ---- | ---- | ---- | ---- | +| resnet50-v1.5 | [vision/classification_and_detection](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection) | tensorflow, onnx, tvm, ncnn | imagenet2012 | edge | +| yolo v11 | [vision/classification_and_detection](https://github.com/mlcommons/inference/tree/master/vision/classification_and_detection/yolo) | pytorch, onnx | COCO safe subset | edge | +| bert | [language/bert](https://github.com/mlcommons/inference/tree/master/language/bert) | tensorflow, pytorch, onnx | squad-1.1 | edge | +| dlrm-v3 | [recommendation/dlrm_v3](https://github.com/mlcommons/inference/tree/master/recommendation/dlrm_v3/pytorch) | pytorch | Synthetic dataset | datacenter | +| 3d-unet | [vision/medical_imaging/3d-unet-kits19](https://github.com/mlcommons/inference/tree/master/vision/medical_imaging/3d-unet-kits19) | pytorch, tensorflow, onnx | KiTS19 | edge | +| stable-diffusion-xl | [text_to_image](https://github.com/mlcommons/inference/tree/master/text_to_image) | pytorch | COCO 2014 | edge | +| Wan2.2-T2V-A14B-Diffusers | [text_to_video](https://github.com/mlcommons/inference/tree/master/text_to_video) | pytorch | COCO 2014 | datacenter | +| llama2-70b | [language/llama2-70b](https://github.com/mlcommons/inference/tree/master/language/llama2-70b) | pytorch | OpenOrca | datacenter | +| rgat | [graph/rgat](https://github.com/mlcommons/inference/tree/master/graph/R-GAT) | pytorch | IGBH | datacenter | +| llama3.1-8b | [language/llama3.1-8b](https://github.com/mlcommons/inference/tree/master/language/llama3.1-8b)| pytorch | CNN-Daily Mail | edge,datacenter | +| deepseek-r1 | [language/deepseek-r1](https://github.com/mlcommons/inference/tree/master/language/deepseek-r1)| pytorch | AIME, MATH500, gpqa, MMLU-Pro, livecodebench(code_generation_lite) | datacenter | +| whisper | [speech2text](https://github.com/mlcommons/inference/tree/master/speech2text)| pytorch | LibriSpeech | edge,datacenter | +| GPT–OSS | [language/gpt-oss-120b](https://github.com/mlcommons/inference/tree/master/language/gpt-oss-120b)| pytorch | mlperf_gpt_oss_performance, mlperf_gpt_oss_accuracy | datacenter | +| VLM (qwen3-vl-235b-a22b) | [multimodal/qwen3-vl](https://github.com/mlcommons/inference/tree/master/multimodal/qwen3-vl)| pytorch | Shopify-product-catalogue | datacenter | +| qwen3.6-27b | [language/edge-agentic](https://github.com/mlcommons/inference/tree/master/language/edge-agentic) | pytorch | BFCL v4 single-turn (accuracy) + recorded agentic-coding replay (performance) | edge | +| e2e-rag (qna, db) | [e2e-rag](https://github.com/mlcommons/inference/tree/master/e2e-rag) | pytorch | FRAMES | datacenter | + +* Framework here is given for the reference implementation. Submitters are free to use their own frameworks to run the benchmark. +* pointpainting, llama3.1-405b and mixtral-8x7b are not part of the v6.1 model list. + ## MLPerf Inference v6.0 (submission deadline February 13, 2026) For submissions, please use the master branch and any commit since the [6.0 seed release](https://github.com/mlcommons/inference/commit/f131a0d29ccae9a967d93ffe96f66b1be3537d3b) although it is best to use the latest commit in the [master branch](https://github.com/mlcommons/inference).