GPU support: separate -gpu / -cuda variant archives for all backends - #11
Open
vackva wants to merge 24 commits into
Open
GPU support: separate -gpu / -cuda variant archives for all backends#11vackva wants to merge 24 commits into
vackva wants to merge 24 commits into
Conversation
Cross-platform GPU plan per backend: CoreML/MPS baked into macOS packages (Phase 1), ORT DirectML on Windows (Phase 2), CUDA variants for ORT+LibTorch on Linux/Windows x64 (Phase 3), ExecuTorch Vulkan delegate as the experimental vendor-agnostic path (Phase 4). TFLite/LiteRT desktop GPU, ROCm, TensorRT and MoltenVK deliberately out of scope.
build-ort.sh passes --use_coreml on the macos leg (static + shared). The EP is inert unless appended, so CPU-only consumers are unchanged. Static consumers must now link CoreML.framework — the smoke CMakeLists documents the link line, and smoke.cpp runs the forward pass a second time with the CoreML EP appended to prove it registers and executes. coreml_proto links protobuf-lite, which the static bundle already ships, so no bundle changes needed.
…Phase 1) USE_MPS=1 on the macos/arm64 leg only, matching upstream's arm64 wheels (MPS-on at deployment target 11.0; runtime-gated to macOS 12.3+). x86_64 stays CPU-only — upstream dropped Intel-mac MPS after 2.2.2 — and torch::mps::is_available() exists in both builds, so the lipo'd universal archive stays link-compatible across slices. The smoke now asserts MPS availability on Apple arm64 and runs a+b on the Metal device to prove kernels dispatch.
BACKENDS_FLAVOR cache var flows preset -> ExternalEngine.cmake -> each engine's stage.sh (extra trailing arg; engines that don't use it ignore it). stage-build gains a libtorch 'channel' input (download.pytorch.org dir: cpu default, cu126 for -cuda legs). The smoke action derives SMOKE_VARIANT from the preset name (*-gpu-* / *-cuda-*) so engine smokes can assert what the package must (and must not) contain.
GPU is always a separate archive (reworks the earlier baked-in CoreML): - accel=coreml (macOS -gpu, static+shared): --use_coreml; the CoreML provider header ships only in this variant, and CPU packages no longer carry it. - accel=dml (Windows -gpu, shared): built from source with --use_dml since Microsoft stopped publishing the DirectML NuGet after 1.24.4; ships onnxruntime.dll + import lib + the nuget-restored Microsoft.AI.DirectML redist DLL + dml_provider_factory.h. - linux-cuda/windows-cuda repackage flavors: upstream -gpu prebuilt with the CUDA provider + bridge shim kept, TensorRT provider dropped, and any DML header removed (headers advertise EPs to the smoke/anira). Smoke runs the forward pass again per shipped EP header (CoreML; DML works on GPU-less runners via WARP).
Default packages are CPU-only again (USE_MPS=0 everywhere; the smoke asserts MPS is ABSENT in default macOS packages). accel=mps builds the separate macOS-arm64-gpu archive; the smoke proves it by running a+b on the Metal device (SMOKE_VARIANT=gpu). -cuda legs repackage the upstream cu126 prebuilts with NVIDIA redistributables stripped (2 GB release asset limit; CUDA 12.x + cuDNN 9 user-provided) and skip the smoke (canRun=0 — they cannot load without the NVIDIA libs).
Default packages are now CPU-only (XNNPACK + optimized kernels), per the GPU-is-always-an-extra-archive rule: CoreML (+MLX on arm64) move from the default macOS packages into accel=coreml -gpu variants. Side effect: the default arm64 package no longer floors at macOS 14 (that came from MLX) — back to 12.0; only the arm64 -gpu variant needs 14+. accel=vulkan (Linux x64 -gpu, experimental) enables the cross-vendor Vulkan delegate: glslc needed at build time only (apt-installed if missing); the loader is dlopen'd via volk so the package gains no hard runtime dependency. Also fixes the MLX metallib bundling check, which keyed on a shell var that was never set.
New legs (vendor.anira.name carries the -gpu/-cuda token, so archive naming/packaging/publish reuse the existing machinery unchanged): - onnx: macOS x86_64/arm64 -gpu (static+shared, CoreML), Windows x64/arm64 -gpu (shared, DirectML), Linux/Windows x64 -cuda (shared) - libtorch: macOS arm64 -gpu (MPS), Linux/Windows x64 -cuda (cu126) - executorch: macOS x86_64/arm64 -gpu (CoreML/+MLX), Linux x64 -gpu (Vulkan, experimental) docs/gpu-support.md rewritten around the variant-only model.
… args Round-1 CI: every from-source -gpu leg (and the ORT -cuda repackages) built CPU-only packages under variant names. Cause: CMake list expansion drops empty elements, so on legs where BACKENDS_URL/BACKENDS_ABIS are empty the trailing flavor arg shifted into the URL slot and stage.sh saw no flavor. The libtorch MPS smoke guard and the Windows DML staging caught it loudly; ORT macOS -gpu / ORT -cuda / executorch -gpu passed FALSELY. Flavor now travels via 'cmake -E env BACKENDS_FLAVOR=...' (positional still wins for direct CLI use), and new false-green guards make every variant self-verify: ORT -gpu must ship an EP provider header, ORT -cuda must ship the CUDA provider lib, executorch -gpu must export its delegate target (libtorch already asserted MPS).
…rG shaderc
Ubuntu 24.04's apt glslc (shaderc 2023.8) predates GL_EXT_integer_dot_product
support and rejects ExecuTorch 1.3.1's int8 shaders at codegen
('#extension : extension not supported'). Probe the actual requirement
(compile a shader that requires the extension) and only when the ambient
glslc fails, add LunarG's noble repo and install its current shaderc.
Not pushed alone — batched with the rest of round-2 fixes so the
in-flight CI round isn't cancelled by the concurrency group.
Round 2 proved the DML build end-to-end (providers_dml compiled, header + DirectML.dll staged); the smoke then failed with C0262002 because ORT's simple device_id path deliberately skips software adapters and the runners have no GPU. Try the hardware path first (covers real machines), and on failure create a WARP D3D12 device explicitly (EnumWarpAdapter -> D3D12CreateDevice -> DMLCreateDevice, resolved from the packaged DirectML.dll at runtime) and append via the OrtDmlApi DML1 entry point — the forward pass then really executes on DirectML. Smoke links d3d12+dxgi for the fallback.
--use_coreml pulls in FP16 -> psimd, whose ancient cmake_minimum_required(<3.5) CMake 4.x refuses — all four macOS -gpu legs died at configure. Same policy floor the libtorch/tflite builders already export; no-op for the existing CPU legs.
…oreML
Upstream CMake bug hit only by static+--use_coreml installs: coreml_proto
is installed but never added to ${PROJECT_NAME}Targets, so generate
aborts ('requires target coreml_proto that is not in any export set' —
both onnxruntime_providers_coreml and the onnxruntime target export-
depend on it). Idempotent perl patch after clone, applied only on the
coreml -gpu legs. Round 3 confirmed everything else: DML legs green
(WARP fallback works), CoreML shared green (policy floor works).
…RFACE
Second layer of the static+CoreML export bug: once coreml_proto is in
the export set (previous fix), its PUBLIC include of
CMAKE_CURRENT_BINARY_DIR is a raw build-dir path, which an installed
export rejects ('INTERFACE_INCLUDE_DIRECTORIES property contains
path: .../build/Release'). $<BUILD_INTERFACE:> keeps the build
identical and drops the path from the export — which nothing consumes;
we ship flat bundled .a's. Round 4 confirmed everything else green:
this was the only remaining error, reported alone in the generate pass.
Surveys CUDA / OpenVINO / CoreML / Linux-GPU / NPU / mobile for all four runtimes, maps everything onto the -gpu/-cuda/-npu token model, and ranks a build order (QNN win-arm64 repackage first). Key 2026 updates folded in: ExecuTorch has an official OpenVINO backend (Intel CPU/iGPU/dGPU/NPU), LiteRT Next has Qualcomm + Intel NPU compiler plugins, ORT's native WebGPU EP now ships manylinux wheels (the eventual DirectML-alternative-on-Linux bet). NOT pushed with intent: any push re-runs all five workflows on the green PR #11.
The vendored include/ set was from ORT_API_VERSION 16 while we ship 1.26.0 binaries — ABI-compatible, but consumers missed ten versions of API (incl. the plugin-EP surface), and the prebuilt-repackaged legs ship upstream's 1.26 headers so our own archives disagreed with each other. Copy the 1.26.0 public set from the pinned source tree (adds onnxruntime_ep_c_api.h, ep-device metadata keys, env config keys).
Final model: CPU everywhere; -gpu = platform-native (CoreML/MPS/MLX on Apple, DirectML on Windows via ORT, Vulkan on Linux via ExecuTorch); -cuda for ORT+LibTorch on Linux/Windows x64. QNN/OpenVINO NPU dropped for now (analysis retained as the revisit reference).
docs/gpu-support.md + docs/accelerator-analysis.md become local-only planning material: removed from the index, docs/ gitignored, and every README link / script-comment reference to them scrubbed. (They remain in branch history; a history rewrite was deliberately not done.)
ios.sh gains a variant arg: gpu builds both slices with --use_coreml (the coreml_proto export patch moved post-clone in build-ort.sh, shared by macOS and iOS coreml builds — deduped from the macos case) and ships the CoreML provider header, with the same curation rule as desktop: the CPU xcframework no longer leaks coreml_provider_factory.h. Consumers of the -gpu xcframework add CoreML.framework.
macOS accel=coreml now also builds the MPS delegate: CoreML and MPS serve different models (ANE-compiled vs direct Metal kernels) and the .pte's export-time partitioning picks, so both in one -gpu archive means any Apple-exported model works — matching iOS. The smoke links the Metal/MPS frameworks when mpsdelegate is exported. ios.sh gains the variant arg: the default xcframework turns CoreML/MPS OFF (CPU-only like every default package — a behavior change vs the previous everything-in xcframework); gpu keeps the upstream preset defaults.
…l) + official iOS delegate frameworks macOS -gpu (4 legs, both arches x kinds): TFLITE_BUILD_METAL_DELEGATE=ON maps to upstream's TFLITE_ENABLE_GPU+TFLITE_ENABLE_METAL (the metal sources live inside the GPU block); ships libmetal_delegate.a + the TFLGpuDelegateCreate header — for the static kind the delegate is also merged into the single bundled lib. Upstream only tests the Metal delegate on iOS, so the macOS legs are experimental tier (we own any macOS-specific breakage). Smoke runs the forward pass through the delegate and false-green-guards the variant on the shipped header. iOS -gpu: repackage also stages Google's official TensorFlowLiteCMetal + TensorFlowLiteCCoreML delegate xcframeworks (pod Metal/CoreML subspecs, same release tarball).
ios job matrixes over ios_variants ('' = CPU, 'gpu'); archive name gains
the variant token (<lib>-<ver>-iOS-gpu-xcframework) and the variant is
passed to engines/<engine>/ios.sh. onnxruntime/tflite/executorch opt in
with '["", "gpu"]'; litert stays CPU-only (its Bazel iOS GPU build is a
deliberate follow-up).
The CUDA-12-generation prebuilts we repackaged ship kernels only up to sm_90 — on RTX 50-series (Blackwell, sm_120) ORT's CUDA EP falls back to CPU and libtorch cu126 has the same gap; that's exactly the GPU a 2026 workstation buyer owns. Switch both backends to one modern NVIDIA stack: ORT gpu_cuda13 release assets + libtorch cu130 (both verified live for our pins; the cu130 zips are also ~40% smaller). User runtime becomes CUDA 13.x + cuDNN 9, driver r580+; oldest supported GPU generation is now Turing (2018) — pre-Turing users fall back to CPU. ORT removes CUDA-12 packages at 1.27 anyway, so this fronts a forced migration.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
GPU acceleration for the anira backends as separate variant archives — default packages stay CPU-only (if a user wants CPU only, they get CPU only; the smokes now even assert the accelerator is absent from default packages).
13 new build legs, keyed off
BACKENDS_FLAVORin the presets; the-gpu/-cudatoken ridesvendor.anira.name, so archive naming, packaging, smoke and publish reuse the existing machinery with no workflow-level changes.-gpuCoreML EP (x86_64+arm64, static+shared,--use_coreml)-gpuDirectML EP (x64+arm64, shared,--use_dmlfrom source — MS stopped the DirectML NuGet at 1.24.4) ·-cuda(x64, repackaged upstream)-cuda(x64, repackaged upstream)-gpuMPS (arm64,USE_MPS=1)-cuda(x64, cu126 repackaged, NVIDIA libs stripped)-cuda(x64, cu126 repackaged, NVIDIA libs stripped)-gpuCoreML (+MLX arm64) — moved out of the default packages; default arm64 floor drops back to macOS 12-gpuVulkan delegate (x64, experimental)Verification
-gpuruns an op on the Metal device. Verified locally on Apple Silicon against an MPS-enabled libtorch (PASS) and the CPU-only assertion correctly rejects an MPS package.Notes
-cudapackages: CUDA 12.x + cuDNN 9 are user-provided (2 GB release-asset limit); libtorch-cudalegs arecanRun=0(can't load without NVIDIA libs).docs/gpu-support.md.https://claude.ai/code/session_01XgDo9VMcE6QEyegtPJx4o9