From 66ddea5d33065af22608314474d40a93cf03eb98 Mon Sep 17 00:00:00 2001 From: Ivan Zatevakhin Date: Sat, 15 Aug 2026 14:06:48 +0100 Subject: [PATCH] feat: add experimental linux backend builds - add deterministic cuda, cutlass, triton aot, and vulkan build planning - preserve static and dynamic linking contracts and package required inputs - add optional cuda and vulkan nix shells with focused validation - document example commands, non-nix setup, and known runtime limitations --- Justfile | 36 +- README.md | 48 ++- flake.nix | 120 +++++- vllm-cpp-sys/Cargo.toml | 8 + vllm-cpp-sys/README.md | 19 +- vllm-cpp-sys/THIRD_PARTY.md | 16 +- vllm-cpp-sys/build.rs | 236 +++++++---- .../licenses/FLASH-ATTENTION-BSD-3-CLAUSE.txt | 29 ++ .../licenses/FLASH-LINEAR-ATTENTION-MIT.txt | 21 + vllm-cpp-sys/src/build_config.rs | 400 ++++++++++++++++++ vllm-cpp-sys/src/build_support.rs | 39 ++ vllm-cpp-sys/tests/build_config.rs | 338 +++++++++++++++ vllm-cpp-sys/tests/build_support.rs | 56 ++- vllm-cpp/Cargo.toml | 4 + vllm-cpp/examples/README.md | 102 +++++ 15 files changed, 1351 insertions(+), 121 deletions(-) create mode 100644 vllm-cpp-sys/licenses/FLASH-ATTENTION-BSD-3-CLAUSE.txt create mode 100644 vllm-cpp-sys/licenses/FLASH-LINEAR-ATTENTION-MIT.txt create mode 100644 vllm-cpp-sys/src/build_config.rs create mode 100644 vllm-cpp-sys/tests/build_config.rs create mode 100644 vllm-cpp/examples/README.md diff --git a/Justfile b/Justfile index e5a4da7..2c11ec6 100644 --- a/Justfile +++ b/Justfile @@ -73,12 +73,28 @@ build-support-test: -o "$temp/build-support-tests" "$temp/build-support-tests" +# Test the pure Linux backend build planner without configuring CMake. +backend-config: + #!/usr/bin/env bash + set -euo pipefail + temp=$(mktemp -d) + trap 'rm -rf "$temp"' EXIT + rustc --edition=2021 --test -D warnings \ + {{ quote(root + "/vllm-cpp-sys/tests/build_config.rs") }} \ + -o "$temp/build-config-tests" + "$temp/build-config-tests" + +# Verify pinned CUDA architecture mappings and vendored Triton AOT inputs. +backend-integrity: + cd {{ quote(root) }} && cmake -P vllm-cpp-sys/vllm.cpp/cmake/CudaArchFeaturesTest.cmake + cd {{ quote(root) }} && bash vllm-cpp-sys/vllm.cpp/scripts/check-triton-aot-drift.sh + # Run the focused C/Rust layout conformance test. layout-test: CARGO_TARGET_DIR="${CARGO_TARGET_DIR:-{{ root }}/target/layout-test}" cargo test --locked -p vllm-cpp-sys --release --test layout -# Run generated-binding, header, and layout conformance checks. -sys: bindings-check header-check build-support-test layout-test +# Run generated-binding, header, layout, and backend configuration conformance checks. +sys: bindings-check header-check build-support-test backend-config backend-integrity layout-test # Test all Linux CPU link modes and the exact shared-library exports. link-modes: @@ -368,10 +384,14 @@ package-test: README.md THIRD_PARTY.md build.rs + licenses/FLASH-ATTENTION-BSD-3-CLAUSE.txt + licenses/FLASH-LINEAR-ATTENTION-MIT.txt wrapper.h src/bindings.rs + src/build_config.rs src/build_support.rs src/lib.rs + tests/build_config.rs tests/build_support.rs tests/layout.c tests/layout.rs @@ -385,11 +405,15 @@ package-test: vllm.cpp/src/capi/engine_handle.h vllm.cpp/src/capi/vllm_c.cpp vllm.cpp/src/vllm/version.cpp + vllm.cpp/src/vt/cuda/triton_aot_vendored/sm_121a/MANIFEST + vllm.cpp/scripts/triton-aot-compile.py + vllm.cpp/triton_kernels/chunk_delta_h.py vllm.cpp/third_party/README.md vllm.cpp/third_party/blake3/LICENSE_A2 vllm.cpp/third_party/blake3/LICENSE_CC0 vllm.cpp/third_party/minja/LICENSE vllm.cpp/third_party/nlohmann/json.hpp + vllm.cpp/third_party/vulkan/vulkan_core.h ) for member in "${required_members[@]}"; do [[ -s $package_root/$member ]] || { @@ -413,16 +437,19 @@ package-test: vllm.cpp/cmake vllm.cpp/include vllm.cpp/src + vllm.cpp/scripts/triton-aot-compile.py + vllm.cpp/triton_kernels vllm.cpp/third_party/README.md vllm.cpp/third_party/blake3 vllm.cpp/third_party/minja vllm.cpp/third_party/nlohmann + vllm.cpp/third_party/vulkan ) diff -u \ <(native_inventory "$repo_root/vllm-cpp-sys" "${native_members[@]}") \ <(native_inventory "$package_root" "${native_members[@]}") diff -u \ - <(printf '%s\n' CMakeLists.txt LICENSE NOTICE cmake include src third_party | LC_ALL=C sort) \ + <(printf '%s\n' CMakeLists.txt LICENSE NOTICE cmake include scripts src third_party triton_kernels | LC_ALL=C sort) \ <(find "$package_root/vllm.cpp" -mindepth 1 -maxdepth 1 -printf '%f\n' | LC_ALL=C sort) denied_members=( Justfile @@ -434,13 +461,10 @@ package-test: vllm.cpp/benchmarks vllm.cpp/docs vllm.cpp/examples - vllm.cpp/scripts vllm.cpp/tests vllm.cpp/tools - vllm.cpp/triton_kernels vllm.cpp/third_party/doctest vllm.cpp/third_party/httplib - vllm.cpp/third_party/vulkan ) for member in "${denied_members[@]}"; do [[ ! -e $package_root/$member ]] || { diff --git a/README.md b/README.md index 153d7aa..396c024 100644 --- a/README.md +++ b/README.md @@ -9,7 +9,7 @@ Rust bindings for [vllm.cpp](https://github.com/mudler/vllm.cpp), organized as: The safe crate provides a cloneable engine API for model loading, blocking completion and streaming, non-blocking concurrent requests, structured output, and raw-JSON chat. An optional `serde` feature adds `serde_json::Value` chat helpers. The sys crate provides checked-in generated FFI declarations with C/Rust layout checks and coverage for all 19 exported C symbols. -Linux x86_64 CPU builds support bundled static, bundled dynamic, system static, and system dynamic linking. vllm.cpp is pinned at `34aedfbe8ed9779697905541a62e2160ccfd9c05`, which exposes C ABI version 10. +Linux x86_64 CPU builds support bundled static, bundled dynamic, system static, and system dynamic linking. Experimental bundled builds also expose Linux x86_64/aarch64 build configuration for CUDA, external CUTLASS, Triton AOT, and Vulkan. These accelerator features are build-only integration surfaces, not runtime-support claims. vllm.cpp is pinned at `34aedfbe8ed9779697905541a62e2160ccfd9c05`, which exposes C ABI version 10. ## Prerequisites @@ -22,10 +22,12 @@ Initial development and testing support Linux CPU builds. They require: - A system linker and C++ standard library. - Just 1.40 or newer for maintainer workflows, plus Git, `jq`, GNU tar, and `curl` for the model fixture recipe. -This repository provides a Nix development shell with the pinned development tools: +This repository provides a Nix development shell with the pinned development tools. Linux also has minimal CUDA and Vulkan shells: ```console nix develop +nix develop .#cuda +nix develop .#vulkan ``` ## Checkout @@ -60,15 +62,7 @@ println!("{:?}", request.wait()?); All streaming callbacks receive copied UTF-8 deltas. Blocking callbacks may borrow stack data; their panics are caught before the C boundary and resumed only after the native call returns. Asynchronous callbacks must be `Send + 'static`, run on a native delivery thread, and report panic as `Error::CallbackPanicked` from `wait`. Waiting for or freeing a request from its own callback thread is prohibited by ABI v10: `wait` returns `Error::RequestCallbackThread`, while drop transfers cleanup to a prestarted reaper that owns the request, callback, and engine until native free/cancel/join completes. Chat methods accept raw OpenAI-compatible request JSON; enable `serde` for `serde_json::Value` request and response helpers. -Run the practical examples with a model directory: - -```console -cargo run -p vllm-cpp --example complete -- -cargo run -p vllm-cpp --example stream -- -cargo run -p vllm-cpp --example concurrent -- -cargo run -p vllm-cpp --example chat -- -cargo run -p vllm-cpp --example structured -- -``` +See [the examples guide](vllm-cpp/examples/README.md) for ordinary Linux and optional Nix setup and commands for every example. ## Build and Test @@ -81,10 +75,38 @@ cargo test --locked -p vllm-cpp --release --features serde just ci ``` -Set `CMAKE_BUILD_PARALLEL_LEVEL` to control native parallelism. The bundled build is deterministic and CPU-only: native tests, examples, the HTTP server, CUDA, Metal, MLX, Vulkan, Triton, and CUTLASS fetching are disabled explicitly. +Set `CMAKE_BUILD_PARALLEL_LEVEL` to control native parallelism. The default bundled build remains deterministic and CPU-only: native tests, examples, the HTTP server, CUDA, Metal, MLX, Vulkan, Triton, and CUTLASS fetching are disabled explicitly. `build.rs` is consumer-only native build/link integration; it does not download dependencies or compile/execute the maintainer layout probe. Ordinary consumers do not need Just, bindgen, or libclang. Normal first-time Cargo dependency resolution may access crates.io; use Cargo's standard `--offline` mode after dependencies are cached. +## Experimental Backend Builds + +Backend features apply to bundled Linux x86_64/aarch64 builds only and are mutually exclusive with `system`; CUDA and Vulkan are also mutually exclusive. Backend features do not enable `bundled`: normal default-feature commands may use `--features cuda`, while `--no-default-features` callers must include it explicitly, for example `--features bundled,cuda`. Use a fresh `CARGO_TARGET_DIR` for every backend and link mode. + +- `cuda` requires `VLLM_CPP_CUDA_ARCHITECTURES` equal to `80`, `86`, `87`, `89`, `90a`, `100a`, `103a`, `110`, `120a`, `121a`, or `120a;121a`. Leave this variable unset when `cuda` is disabled, including CPU and system builds. +- `cuda-cutlass` implies `cuda`, requires an explicit canonical `VLLM_CPP_CUTLASS_DIR` containing CUTLASS >=4.5.0, disables fetching, and rejects `103a` and `110`. Plain CUDA uses a nonexistent sentinel CUTLASS root so an ambient checkout cannot alter the build. +- `triton-aot` implies `cuda`, enables only checked-in AOT artifacts for one of `80`, `86`, `89`, `90a`, `100a`, or `121a`, and forces regeneration off. +- `vulkan` uses packaged Khronos headers and checked-in SPIR-V. It does not link a Vulkan SDK library; the native library opens the runtime loader dynamically. + +For example: + +```console +nix develop .#cuda +VLLM_CPP_CUDA_ARCHITECTURES=120a \ + CARGO_TARGET_DIR=target/cuda-static \ + cargo build --locked --release --features cuda +VLLM_CPP_CUDA_ARCHITECTURES=120a \ + CARGO_TARGET_DIR=target/cuda-dynamic \ + cargo build --locked --release --features cuda,dynamic-link + +nix develop .#vulkan +CARGO_TARGET_DIR=target/vulkan-static cargo build --locked --release --features vulkan +``` + +Static CUDA links the exact `cudart`, `cublasLt`, and, for Triton, CUDA driver locations selected by CMake. Dynamic builds rely on `libvllm.so` `DT_NEEDED` entries instead of repeating those transitive Cargo links; deploy the shared library and toolkit libraries through normal loader paths. + +Compilation does not establish runtime correctness. Known native evidence blockers remain: CUDA teardown can SIGSEGV after otherwise successful tests; CUDA bf16 testing has a numerical tolerance failure; CUTLASS concurrent output differs from the non-concurrent path; Vulkan runtime coverage is incomplete. No runtime support is claimed here. + ## Test Model and Sanitizers Model-backed tests use Apache-2.0 `Qwen/Qwen3-0.6B` at pinned revision `c1899de289a04d12100db370d81485cdf75e47ca`. Download or reuse the cache and verify every file, then run exactly 14 blocking and request-lifecycle model tests serially: @@ -129,7 +151,7 @@ The package gate preserves the sys crate inventory, tests the extracted sys crat ## Support -The supported target is native Linux x86_64 CPU. Maintainer tests cover the four bundled/system static/dynamic link modes plus bundled blocking and concurrent request inference with the pinned Qwen fixture. Sanitizer evidence covers native ASan/UBSan/leak detection and selected native-only GCC TSan lifecycle paths as described above. Other operating systems, architectures, and accelerator builds are not supported by this Rust build. +The supported runtime target is native Linux x86_64 CPU. Maintainer tests cover the four bundled/system static/dynamic CPU link modes plus bundled blocking and concurrent request inference with the pinned Qwen fixture. Sanitizer evidence covers native ASan/UBSan/leak detection and selected native-only GCC TSan lifecycle paths as described above. Linux CUDA/CUTLASS/Triton/Vulkan features remain experimental build-only surfaces with the limitations listed above; Apple and other accelerator targets are out of scope. ## Licensing and Affiliation diff --git a/flake.nix b/flake.nix index 26ca83f..f22ae48 100644 --- a/flake.nix +++ b/flake.nix @@ -16,28 +16,110 @@ pkgs = import inputs.nixpkgs { inherit system overlays; }; + cudaPkgs = import inputs.nixpkgs { + inherit system overlays; + config.allowUnfree = true; + }; rustToolchain = pkgs.rust-bin.fromRustupToolchainFile ./rust-toolchain.toml; in { - devShells.default = pkgs.mkShell { - packages = [ - rustToolchain - pkgs.cmake - pkgs.git - pkgs.just - pkgs.jq - pkgs.ninja - pkgs.pkg-config - pkgs.gnutar - pkgs.llvmPackages.clang - pkgs.llvmPackages.bintools - pkgs.rust-bindgen - ]; - - shellHook = '' - export PS1="(dev:vllm-cpp-rs) $PS1" - ''; - }; + devShells = + { + default = pkgs.mkShell { + packages = [ + rustToolchain + pkgs.cmake + pkgs.git + pkgs.just + pkgs.jq + pkgs.ninja + pkgs.pkg-config + pkgs.gnutar + pkgs.llvmPackages.clang + pkgs.llvmPackages.bintools + pkgs.rust-bindgen + ]; + + shellHook = '' + export PS1="(dev:vllm-cpp-rs) $PS1" + ''; + }; + } + // pkgs.lib.optionalAttrs + (builtins.elem system [ + "x86_64-linux" + "aarch64-linux" + ]) { + cuda = let + toolkit = cudaPkgs.cudaPackages.cudatoolkit; + cutlass = cudaPkgs.cudaPackages.cutlass; + in + pkgs.mkShell { + packages = [ + rustToolchain + pkgs.cmake + pkgs.git + pkgs.just + pkgs.jq + pkgs.ninja + pkgs.pkg-config + pkgs.gnutar + pkgs.llvmPackages.clang + pkgs.llvmPackages.bintools + pkgs.rust-bindgen + toolkit + cutlass + ]; + + shellHook = '' + export PS1="(cuda:vllm-cpp-rs) $PS1" + export CUDA_PATH="${toolkit}" + export CUDA_HOME="$CUDA_PATH" + export CUDAToolkit_ROOT="$CUDA_PATH" + export VLLM_CPP_CUTLASS_DIR="${cutlass.src}" + if [ -d /run/opengl-driver/lib ]; then + export LD_LIBRARY_PATH="/run/opengl-driver/lib:${toolkit}/lib''${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" + else + export LD_LIBRARY_PATH="${toolkit}/lib''${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" + fi + ''; + }; + + vulkan = let + vulkanLibraryPath = pkgs.lib.makeLibraryPath [ + pkgs.vulkan-loader + pkgs.vulkan-validation-layers + ]; + in + pkgs.mkShell { + packages = [ + rustToolchain + pkgs.cmake + pkgs.git + pkgs.just + pkgs.jq + pkgs.ninja + pkgs.pkg-config + pkgs.gnutar + pkgs.llvmPackages.clang + pkgs.llvmPackages.bintools + pkgs.rust-bindgen + pkgs.vulkan-tools + pkgs.vulkan-loader + pkgs.vulkan-validation-layers + ]; + + shellHook = '' + export PS1="(vulkan:vllm-cpp-rs) $PS1" + if [ -d /run/opengl-driver/lib ]; then + export LD_LIBRARY_PATH="/run/opengl-driver/lib:${vulkanLibraryPath}''${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" + else + export LD_LIBRARY_PATH="${vulkanLibraryPath}''${LD_LIBRARY_PATH:+:$LD_LIBRARY_PATH}" + fi + export VK_LAYER_PATH="${pkgs.vulkan-validation-layers}/share/vulkan/explicit_layer.d" + ''; + }; + }; }; }; } diff --git a/vllm-cpp-sys/Cargo.toml b/vllm-cpp-sys/Cargo.toml index 98082c6..a657b19 100644 --- a/vllm-cpp-sys/Cargo.toml +++ b/vllm-cpp-sys/Cargo.toml @@ -19,13 +19,17 @@ include = [ "/LICENSE-MIT", "/NOTICE", "/THIRD_PARTY.md", + "/licenses/**", "/vllm.cpp/CMakeLists.txt", "/vllm.cpp/cmake/**", "/vllm.cpp/include/**", "/vllm.cpp/src/**", + "/vllm.cpp/triton_kernels/**", + "/vllm.cpp/scripts/triton-aot-compile.py", "/vllm.cpp/third_party/blake3/**", "/vllm.cpp/third_party/minja/**", "/vllm.cpp/third_party/nlohmann/**", + "/vllm.cpp/third_party/vulkan/**", "/vllm.cpp/LICENSE", "/vllm.cpp/NOTICE", "/vllm.cpp/third_party/README.md", @@ -36,6 +40,10 @@ default = ["bundled"] bundled = [] system = [] dynamic-link = [] +cuda = [] +cuda-cutlass = ["cuda"] +triton-aot = ["cuda"] +vulkan = [] [build-dependencies] cmake = "0.1.58" diff --git a/vllm-cpp-sys/README.md b/vllm-cpp-sys/README.md index f679e9c..d8705ce 100644 --- a/vllm-cpp-sys/README.md +++ b/vllm-cpp-sys/README.md @@ -10,7 +10,7 @@ This crate exposes generated unsafe functions that mirror the stable C API. Appl - `dynamic-link` makes either source mode link `libvllm` dynamically. - `system` disables the bundled build and links a caller-provided installation. Disable default features when selecting it. -`bundled` and `system` are mutually exclusive. Initial support is limited to native Linux CPU builds. +`bundled` and `system` are mutually exclusive. CPU runtime support is limited to native Linux x86_64; Linux x86_64/aarch64 accelerator features below are experimental build-only configuration. System mode requires `VLLM_CPP_ROOT`, whose prefix must contain `include/vllm.h` plus a `lib` or `lib64` directory. `VLLM_CPP_LIB_DIR` can override the vllm library directory. Consumer builds validate that the selected system header exists but do not compare its layout. The maintainer integration test compiles its C probe at test runtime against that header, compares its C layouts with the generated Rust declarations, and the runtime test requires ABI version 10. @@ -18,6 +18,21 @@ Upstream's normal CMake install provides `libvllm` and `vllm.h` but does not ins `dynamic-link` does not copy or package the shared library. Tests and applications must install `libvllm` in a loader-visible location or configure `LD_LIBRARY_PATH`, rpath, or another loader search path. +## Backend Features + +Feature selection is build configuration, not runtime or hardware evidence. Accelerator features are bundled-only but do not enable `bundled`: normal default-feature commands may use `--features cuda`, while `--no-default-features` callers must use, for example, `--features bundled,cuda`. `cuda` conflicts with `vulkan`, and host `VLLM_CPP_SANITIZE` settings conflict with CUDA. + +| Feature | Build contract | Native linking | +|---|---|---| +| `cuda` | Linux x86_64/aarch64; requires exact `VLLM_CPP_CUDA_ARCHITECTURES` | Static mode links CMake's exact `CUDA_CUDART` and `CUDA_cublasLt_LIBRARY` results; dynamic mode relies on `libvllm.so` dependencies | +| `cuda-cutlass` | Implies CUDA; requires canonical `VLLM_CPP_CUTLASS_DIR` with CUTLASS >=4.5.0; fetch is OFF; `103a`/`110` rejected | Header-only external input | +| `triton-aot` | Implies CUDA; checked-in single-architecture artifacts only; regeneration is OFF | Static mode also links CMake's exact `CUDA_cuda_driver_LIBRARY`; dynamic mode relies on `libvllm.so` | +| `vulkan` | Linux x86_64/aarch64; packaged Khronos headers and checked-in SPIR-V | No Vulkan SDK link; runtime loader uses `dlopen` | + +CUDA architectures are exactly `80`, `86`, `87`, `89`, `90a`, `100a`, `103a`, `110`, `120a`, `121a`, or `120a;121a`. Triton accepts only `80`, `86`, `89`, `90a`, `100a`, or `121a`. Plain CUDA passes a nonexistent CUTLASS root to CMake so ambient source cannot silently change the build. Use separate `CARGO_TARGET_DIR` values for each backend/link combination. + +These features do not claim runtime support. Known native blockers remain: a CUDA teardown SIGSEGV after otherwise successful tests, a CUDA bf16 numerical tolerance failure, CUTLASS concurrent output differences, and incomplete Vulkan runtime coverage. Metal and MLX are not exposed by this crate. + ## Generated Bindings The bundled source is pinned to commit `34aedfbe8ed9779697905541a62e2160ccfd9c05` and exposes C ABI version 10. Bindings are generated with bindgen 0.72.1 from `wrapper.h`, which includes `vllm.cpp/include/vllm.h`, and committed to `src/bindings.rs`. Maintainers use Just 1.40 or newer from the repository root: @@ -28,7 +43,7 @@ just sys just link-modes ``` -The conformance gate verifies the generated output, C and C++ header compatibility, C/Rust layout, the exact 19-symbol export set, and compile-time and runtime ABI 10. CI separately runs bundled static/dynamic and fixture-backed system static/dynamic tests; dynamic tests set the required loader path. `build.rs` only performs consumer native build/link integration and does not compile or execute the layout probe. `tests/layout.rs` compiles and executes `tests/layout.c` with the bundled header or `VLLM_CPP_ROOT/include/vllm.h` at test runtime using the Rust standard library. Native Linux CPU is the supported target; cross-compiling that integration test is unsupported. +The conformance gate verifies the generated output, C and C++ header compatibility, C/Rust layout, the exact 19-symbol export set, pure backend plans/cache parsing, pinned CUDA architecture mappings, Triton AOT drift, and compile-time/runtime ABI 10. CI separately runs bundled static/dynamic and fixture-backed system static/dynamic CPU tests; dynamic tests set the required loader path. `build.rs` only performs consumer native build/link integration and does not compile or execute the layout probe. `tests/layout.rs` compiles and executes `tests/layout.c` with the bundled header or `VLLM_CPP_ROOT/include/vllm.h` at test runtime using the Rust standard library. Native Linux CPU is the supported runtime target; cross-compiling that integration test is unsupported. The Rust crate is dual-licensed under MIT or Apache-2.0. The bundled vllm.cpp source retains its upstream Apache-2.0 license and notices. diff --git a/vllm-cpp-sys/THIRD_PARTY.md b/vllm-cpp-sys/THIRD_PARTY.md index 9942d0f..aea57ab 100644 --- a/vllm-cpp-sys/THIRD_PARTY.md +++ b/vllm-cpp-sys/THIRD_PARTY.md @@ -2,11 +2,17 @@ `vllm-cpp-sys` redistributes the pinned vllm.cpp source required for offline native builds. vllm.cpp is Apache-2.0 and incorporates or vendors components under their own licenses. -Canonical attribution and license information is included in: +| Component | Packaged source | Provenance | License text | +|---|---|---|---| +| vllm.cpp | `vllm.cpp/**` | vllm.cpp commit `34aedfbe8ed9779697905541a62e2160ccfd9c05` | `vllm.cpp/LICENSE`, `vllm.cpp/NOTICE` | +| BLAKE3 C reference | `vllm.cpp/third_party/blake3/**` | BLAKE3 1.5.5, commit `81f772a` | `vllm.cpp/third_party/blake3/LICENSE_A2`, `vllm.cpp/third_party/blake3/LICENSE_CC0` | +| google/minja | `vllm.cpp/third_party/minja/**` | minja commit `021c229` | `vllm.cpp/third_party/minja/LICENSE` | +| Vulkan-Headers | `vllm.cpp/third_party/vulkan/**` | Vulkan SDK 1.4.328.1, generated Khronos headers | Apache-2.0 in each generated header and `vllm.cpp/LICENSE` | +| FlashAttention-2 slice | `vllm.cpp/src/vt/cuda/flash_attn/**` | `vllm-project/flash-attention` commit `2c839c33`, as recorded by the vllm.cpp import | `licenses/FLASH-ATTENTION-BSD-3-CLAUSE.txt` | +| Marlin / GPTQ-Marlin slice | `vllm.cpp/src/vt/cuda/marlin/**` | vLLM commit `e24d1b24`, with retained file notices and vllm.cpp adapter changes | Apache-2.0 in `vllm.cpp/LICENSE` and retained source notices | +| Flash Linear Attention Triton kernels | `vllm.cpp/triton_kernels/**`, generated AOT files under `vllm.cpp/src/vt/cuda/triton_aot_vendored/**` | FLA source ported through vLLM 0.24.0; exact source and artifact hashes are pinned in each AOT `MANIFEST` | `licenses/FLASH-LINEAR-ATTENTION-MIT.txt` | +| nlohmann/json | `vllm.cpp/third_party/nlohmann/**` | nlohmann/json 3.12.0 | MIT notice retained in `json.hpp` | -- `NOTICE` -- `vllm.cpp/NOTICE` -- `vllm.cpp/third_party/README.md` -- License notices retained in vendored source files and directories +The package excludes upstream tests, fixtures, benchmarks, models, external SDKs, and build output. It contains only native source/build inputs and required licenses/notices. vllm.cpp is an independent community project and is not affiliated with or endorsed by the vLLM project, the PyTorch Foundation, or the Linux Foundation. diff --git a/vllm-cpp-sys/build.rs b/vllm-cpp-sys/build.rs index 44c5465..fe08bcb 100644 --- a/vllm-cpp-sys/build.rs +++ b/vllm-cpp-sys/build.rs @@ -1,3 +1,5 @@ +#[path = "src/build_config.rs"] +mod build_config; #[path = "src/build_support.rs"] mod build_support; @@ -5,7 +7,12 @@ use std::env; use std::fs; use std::path::{Path, PathBuf}; -use build_support::{find_installed_library_dir, require_library_file, shared_library_name}; +use build_config::{ + BuildPlan, CudaComponent, Environment, Features, FsProbe, Inputs, LinkRequirement, Target, +}; +use build_support::{ + cmake_cache_path, find_installed_library_dir, require_library_file, shared_library_name, +}; const RERUN_ENV: &[&str] = &[ "CMAKE_BUILD_PARALLEL_LEVEL", @@ -16,7 +23,12 @@ const RERUN_ENV: &[&str] = &[ "VLLM_CPP_ROOT", "VLLM_CPP_LIB_DIR", "VLLM_CPP_BLAKE3_LIB_DIR", + "VLLM_CPP_CUDA_ARCHITECTURES", + "VLLM_CPP_CUTLASS_DIR", "VLLM_CPP_SANITIZE", + "CUDA_PATH", + "CUDA_HOME", + "CUDAToolkit_ROOT", "CC", "CFLAGS", "CXX", @@ -25,13 +37,19 @@ const RERUN_ENV: &[&str] = &[ fn main() { for path in [ + "build.rs", + "src/build_config.rs", + "src/build_support.rs", "vllm.cpp/CMakeLists.txt", "vllm.cpp/cmake", "vllm.cpp/include/vllm.h", "vllm.cpp/src", + "vllm.cpp/triton_kernels", + "vllm.cpp/scripts/triton-aot-compile.py", "vllm.cpp/third_party/blake3", "vllm.cpp/third_party/minja", "vllm.cpp/third_party/nlohmann", + "vllm.cpp/third_party/vulkan", ] { println!("cargo:rerun-if-changed={path}"); } @@ -43,51 +61,64 @@ fn main() { return; } - let bundled = cfg!(feature = "bundled"); - let system = cfg!(feature = "system"); - match (bundled, system) { - (true, true) => panic!("features `bundled` and `system` are mutually exclusive"), - (false, false) => panic!("enable exactly one of the `bundled` or `system` features"), - _ => {} - } - - let target_os = env::var("CARGO_CFG_TARGET_OS").expect("cargo sets target OS"); - if target_os != "linux" { - panic!("linking is implemented only for Linux, not {target_os}"); - } - - let sanitizer = sanitizer_config(bundled); - let system_root = system.then(validate_system_root); - if bundled { - build_bundled(sanitizer.as_deref()); + let inputs = build_inputs(); + let plan = build_config::plan(&inputs, &FsProbe).unwrap_or_else(|error| panic!("{error}")); + let sanitizer = sanitizer_config(inputs.features.bundled); + let system_root = inputs.features.system.then(validate_system_root); + let cmake_cache = if inputs.features.bundled { + Some(build_bundled(&plan)) } else { link_system(system_root.as_deref().expect("system root was validated")); - } + None + }; + emit_link_requirements(&plan, cmake_cache.as_deref()); link_sanitizer_runtimes(sanitizer.as_deref()); } -fn build_bundled(sanitizer: Option<&str>) { +fn build_inputs() -> Inputs { + Inputs { + features: Features { + bundled: cfg!(feature = "bundled"), + system: cfg!(feature = "system"), + dynamic_link: cfg!(feature = "dynamic-link"), + cuda: cfg!(feature = "cuda"), + cuda_cutlass: cfg!(feature = "cuda-cutlass"), + triton_aot: cfg!(feature = "triton-aot"), + vulkan: cfg!(feature = "vulkan"), + }, + target: Target { + triple: required_env("TARGET"), + os: required_env("CARGO_CFG_TARGET_OS"), + arch: required_env("CARGO_CFG_TARGET_ARCH"), + }, + environment: Environment { + cuda_architectures: env::var("VLLM_CPP_CUDA_ARCHITECTURES").ok(), + cutlass_dir: env::var_os("VLLM_CPP_CUTLASS_DIR").map(PathBuf::from), + sanitizer: env::var("VLLM_CPP_SANITIZE").ok(), + }, + } +} + +fn build_bundled(plan: &BuildPlan) -> PathBuf { let source = Path::new("vllm.cpp"); if !source.join("CMakeLists.txt").is_file() { - panic!( - "vllm.cpp source is missing; initialize it with `git submodule update --init --recursive`" + config_error( + "vllm.cpp source is missing; initialize it with `git submodule update --init --recursive`", ); } + let out_dir = PathBuf::from(required_env("OUT_DIR")); let mut config = cmake::Config::new(source); - config - .profile("Release") - .define("VLLM_CPP_BUILD_TESTS", "OFF") - .define("VLLM_CPP_BUILD_EXAMPLES", "OFF") - .define("VLLM_CPP_SERVER", "OFF") - .define("VLLM_CPP_CUDA", "OFF") - .define("VLLM_CPP_METAL", "OFF") - .define("VLLM_CPP_VULKAN", "OFF") - .define("VLLM_CPP_MLX", "OFF") - .define("VLLM_CPP_TRITON", "OFF") - .define("VLLM_CPP_TRITON_REGEN", "OFF") - .define("VLLM_CPP_CUTLASS_FETCH", "OFF") - .define("VLLM_CPP_SANITIZE", sanitizer.unwrap_or("OFF")); + config.profile("Release"); + for (name, value) in &plan.cmake_defines { + config.define(name, value); + } + if !cfg!(feature = "cuda-cutlass") { + config.define( + "VLLM_CPP_CUTLASS_DIR", + out_dir.join("disabled-cutlass-feature"), + ); + } let dynamic_link = cfg!(feature = "dynamic-link"); let vllm_artifact = if dynamic_link { @@ -97,8 +128,12 @@ fn build_bundled(sanitizer: Option<&str>) { }; let install = config.build(); - let installed_lib_dir = find_installed_library_dir(&install, &vllm_artifact) - .unwrap_or_else(|error| panic!("failed to select bundled library directory: {error}")); + let installed_lib_dir = + find_installed_library_dir(&install, &vllm_artifact).unwrap_or_else(|error| { + config_error(format!( + "failed to select bundled library directory: {error}" + )) + }); let vllm = require_library_file( &installed_lib_dir, &vllm_artifact, @@ -108,7 +143,7 @@ fn build_bundled(sanitizer: Option<&str>) { "bundled static vllm" }, ) - .unwrap_or_else(|error| panic!("{error}")); + .unwrap_or_else(|error| config_error(error)); println!("cargo:rerun-if-changed={}", vllm.display()); println!( "cargo:rustc-link-search=native={}", @@ -117,33 +152,44 @@ fn build_bundled(sanitizer: Option<&str>) { if dynamic_link { println!("cargo:rustc-link-lib=dylib=vllm"); - return; + } else { + let blake3 = find_unique_file( + &out_dir.join("build"), + static_library_name("blake3_vendored"), + ) + .unwrap_or_else(|error| { + config_error(format!( + "failed to locate blake3_vendored build archive: {error}" + )) + }); + println!("cargo:rerun-if-changed={}", blake3.display()); + println!( + "cargo:rustc-link-search=native={}", + blake3.parent().expect("library has a parent").display() + ); + println!("cargo:rustc-link-lib=static:+whole-archive=vllm"); + println!("cargo:rustc-link-lib=static=blake3_vendored"); } - let out_dir = PathBuf::from(env::var_os("OUT_DIR").expect("cargo always sets OUT_DIR")); - let blake3 = find_unique_file( - &out_dir.join("build"), - static_library_name("blake3_vendored"), - ) - .unwrap_or_else(|error| panic!("failed to locate blake3_vendored build archive: {error}")); - println!("cargo:rerun-if-changed={}", blake3.display()); - println!( - "cargo:rustc-link-search=native={}", - blake3.parent().expect("library has a parent").display() - ); - println!("cargo:rustc-link-lib=static:+whole-archive=vllm"); - println!("cargo:rustc-link-lib=static=blake3_vendored"); - link_platform_dependencies(); + let cache = out_dir.join("build/CMakeCache.txt"); + if !cache.is_file() { + config_error(format!( + "CMake did not produce the expected cache at {}", + cache.display() + )); + } + println!("cargo:rerun-if-changed={}", cache.display()); + cache } fn validate_system_root() -> PathBuf { let root = required_path("VLLM_CPP_ROOT"); let header = root.join("include/vllm.h"); if !header.is_file() { - panic!( + config_error(format!( "VLLM_CPP_ROOT must contain include/vllm.h; missing {}", header.display() - ); + )); } println!("cargo:rerun-if-changed={}", header.display()); root @@ -166,7 +212,7 @@ fn link_system(root: &Path) { "system static vllm" }, ) - .unwrap_or_else(|error| panic!("{error}")); + .unwrap_or_else(|error| config_error(error)); println!("cargo:rerun-if-changed={}", vllm.display()); if dynamic_link { @@ -179,17 +225,17 @@ fn link_system(root: &Path) { .map(PathBuf::from) .unwrap_or_else(|| lib_dir.clone()); if !blake3_lib_dir.is_dir() { - panic!( + config_error(format!( "VLLM_CPP_BLAKE3_LIB_DIR does not exist: {}", blake3_lib_dir.display() - ); + )); } let blake3 = require_library_file( &blake3_lib_dir, &static_library_name("blake3_vendored"), "system static blake3_vendored; provision it separately and set VLLM_CPP_BLAKE3_LIB_DIR, or use `system,dynamic-link`", ) - .unwrap_or_else(|error| panic!("{error}")); + .unwrap_or_else(|error| config_error(error)); println!("cargo:rerun-if-changed={}", blake3.display()); println!( "cargo:rustc-link-search=native={}", @@ -200,43 +246,85 @@ fn link_system(root: &Path) { } println!("cargo:rustc-link-lib=static:+whole-archive=vllm"); println!("cargo:rustc-link-lib=static=blake3_vendored"); - link_platform_dependencies(); } fn system_library_dir(root: &Path, expected_artifact: &str) -> PathBuf { if let Some(lib_dir) = env::var_os("VLLM_CPP_LIB_DIR").map(PathBuf::from) { if !lib_dir.is_dir() { - panic!("VLLM_CPP_LIB_DIR does not exist: {}", lib_dir.display()); + config_error(format!( + "VLLM_CPP_LIB_DIR does not exist: {}", + lib_dir.display() + )); } require_library_file(&lib_dir, expected_artifact, "VLLM_CPP_LIB_DIR override") - .unwrap_or_else(|error| panic!("{error}")); + .unwrap_or_else(|error| config_error(error)); return lib_dir; } find_installed_library_dir(root, expected_artifact).unwrap_or_else(|error| { - panic!( + config_error(format!( "failed to locate {expected_artifact} under {}: {error}", root.display() - ) + )) }) } +fn emit_link_requirements(plan: &BuildPlan, cmake_cache: Option<&Path>) { + for requirement in &plan.link_requirements { + match requirement { + LinkRequirement::Library(name) => { + println!("cargo:rustc-link-lib=dylib={name}"); + } + LinkRequirement::CudaToolkit(component) => { + let cache = cmake_cache.unwrap_or_else(|| { + config_error( + "internal error: a CUDA link requirement has no bundled CMake cache", + ) + }); + link_cuda_component(cache, *component); + } + } + } +} + +fn link_cuda_component(cache: &Path, component: CudaComponent) { + let library = cmake_cache_path(cache, component.cmake_cache_key()).unwrap_or_else(|error| { + config_error(format!( + "failed to resolve CUDA component {} from CMake's CUDAToolkit result: {error}", + component.cargo_library() + )) + }); + if !library.is_file() { + config_error(format!( + "CMake selected CUDA component {} at {}, but that file does not exist", + component.cargo_library(), + library.display() + )); + } + println!("cargo:rerun-if-changed={}", library.display()); + println!( + "cargo:rustc-link-search=native={}", + library.parent().expect("library has a parent").display() + ); + println!("cargo:rustc-link-lib=dylib={}", component.cargo_library()); +} + fn sanitizer_config(bundled: bool) -> Option { let value = env::var_os("VLLM_CPP_SANITIZE")?; let value = value .into_string() - .unwrap_or_else(|_| panic!("VLLM_CPP_SANITIZE must be valid UTF-8")); + .unwrap_or_else(|_| config_error("VLLM_CPP_SANITIZE must be valid UTF-8")); if value == "OFF" { return None; } if !bundled { - panic!("VLLM_CPP_SANITIZE is supported only for bundled builds"); + config_error("VLLM_CPP_SANITIZE is supported only for bundled builds"); } match value.as_str() { "address" | "undefined" | "address,undefined" | "thread" => Some(value), - _ => panic!( + _ => config_error(format!( "unsupported VLLM_CPP_SANITIZE value `{value}`; expected OFF, address, undefined, address,undefined, or thread" - ), + )), } } @@ -255,16 +343,18 @@ fn link_sanitizer_runtimes(sanitizer: Option<&str>) { } } -fn link_platform_dependencies() { - println!("cargo:rustc-link-lib=dylib=stdc++"); - println!("cargo:rustc-link-lib=dylib=pthread"); - println!("cargo:rustc-link-lib=dylib=dl"); +fn required_env(name: &str) -> String { + env::var(name).unwrap_or_else(|_| config_error(format!("Cargo did not set required {name}"))) } fn required_path(name: &str) -> PathBuf { env::var_os(name) .map(PathBuf::from) - .unwrap_or_else(|| panic!("{name} must be set when the `system` feature is enabled")) + .unwrap_or_else(|| config_error(format!("{name} must be set for `system` mode"))) +} + +fn config_error(message: impl AsRef) -> ! { + panic!("{} {}", build_config::ERROR_PREFIX, message.as_ref()) } fn static_library_name(stem: &str) -> String { diff --git a/vllm-cpp-sys/licenses/FLASH-ATTENTION-BSD-3-CLAUSE.txt b/vllm-cpp-sys/licenses/FLASH-ATTENTION-BSD-3-CLAUSE.txt new file mode 100644 index 0000000..5860e4b --- /dev/null +++ b/vllm-cpp-sys/licenses/FLASH-ATTENTION-BSD-3-CLAUSE.txt @@ -0,0 +1,29 @@ +BSD 3-Clause License + +Copyright (c) 2022, the respective contributors, as shown by the AUTHORS file. +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +* Redistributions of source code must retain the above copyright notice, this + list of conditions and the following disclaimer. + +* Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +* Neither the name of the copyright holder nor the names of its + contributors may be used to endorse or promote products derived from + this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE +FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL +DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR +SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER +CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, +OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE +OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. diff --git a/vllm-cpp-sys/licenses/FLASH-LINEAR-ATTENTION-MIT.txt b/vllm-cpp-sys/licenses/FLASH-LINEAR-ATTENTION-MIT.txt new file mode 100644 index 0000000..7ced501 --- /dev/null +++ b/vllm-cpp-sys/licenses/FLASH-LINEAR-ATTENTION-MIT.txt @@ -0,0 +1,21 @@ +MIT License + +Copyright (c) 2023-2026 Songlin Yang, Yu Zhang, Zhiyuan Li + +Permission is hereby granted, free of charge, to any person obtaining a copy +of this software and associated documentation files (the "Software"), to deal +in the Software without restriction, including without limitation the rights +to use, copy, modify, merge, publish, distribute, sublicense, and/or sell +copies of the Software, and to permit persons to whom the Software is +furnished to do so, subject to the following conditions: + +The above copyright notice and this permission notice shall be included in all +copies or substantial portions of the Software. + +THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR +IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, +FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE +AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER +LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, +OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE +SOFTWARE. diff --git a/vllm-cpp-sys/src/build_config.rs b/vllm-cpp-sys/src/build_config.rs new file mode 100644 index 0000000..75cc19a --- /dev/null +++ b/vllm-cpp-sys/src/build_config.rs @@ -0,0 +1,400 @@ +use std::fmt; +use std::fs; +use std::path::{Path, PathBuf}; + +pub const ERROR_PREFIX: &str = "vllm-cpp-sys build configuration error:"; + +const CUDA_ARCHITECTURES: &[&str] = &[ + "80", + "86", + "87", + "89", + "90a", + "100a", + "103a", + "110", + "120a", + "121a", + "120a;121a", +]; +const TRITON_ARCHITECTURES: &[&str] = &["80", "86", "89", "90a", "100a", "121a"]; + +#[derive(Clone, Debug, Default, Eq, PartialEq)] +pub struct Features { + pub bundled: bool, + pub system: bool, + pub dynamic_link: bool, + pub cuda: bool, + pub cuda_cutlass: bool, + pub triton_aot: bool, + pub vulkan: bool, +} + +impl Features { + fn has_backend(&self) -> bool { + self.cuda || self.cuda_cutlass || self.triton_aot || self.vulkan + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Target { + pub triple: String, + pub os: String, + pub arch: String, +} + +#[derive(Clone, Debug, Default, Eq, PartialEq)] +pub struct Environment { + pub cuda_architectures: Option, + pub cutlass_dir: Option, + pub sanitizer: Option, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct Inputs { + pub features: Features, + pub target: Target, + pub environment: Environment, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub enum LinkRequirement { + Library(&'static str), + CudaToolkit(CudaComponent), +} + +#[derive(Clone, Copy, Debug, Eq, PartialEq)] +pub enum CudaComponent { + Runtime, + BlasLt, + Driver, +} + +impl CudaComponent { + pub fn cmake_cache_key(self) -> &'static str { + match self { + Self::Runtime => "CUDA_CUDART", + Self::BlasLt => "CUDA_cublasLt_LIBRARY", + Self::Driver => "CUDA_cuda_driver_LIBRARY", + } + } + + pub fn cargo_library(self) -> &'static str { + match self { + Self::Runtime => "cudart", + Self::BlasLt => "cublasLt", + Self::Driver => "cuda", + } + } +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct BuildPlan { + pub cmake_defines: Vec<(&'static str, String)>, + pub link_requirements: Vec, +} + +#[derive(Clone, Debug, Eq, PartialEq)] +pub struct ConfigError(String); + +impl ConfigError { + fn new(message: impl Into) -> Self { + Self(message.into()) + } +} + +impl fmt::Display for ConfigError { + fn fmt(&self, formatter: &mut fmt::Formatter<'_>) -> fmt::Result { + write!(formatter, "{ERROR_PREFIX} {}", self.0) + } +} + +impl std::error::Error for ConfigError {} + +pub trait PathProbe { + fn is_file(&self, path: &Path) -> bool; + fn is_dir(&self, path: &Path) -> bool; + fn canonicalize(&self, path: &Path) -> Result; + fn read_to_string(&self, path: &Path) -> Result; +} + +pub struct FsProbe; + +impl PathProbe for FsProbe { + fn is_file(&self, path: &Path) -> bool { + path.is_file() + } + + fn is_dir(&self, path: &Path) -> bool { + path.is_dir() + } + + fn canonicalize(&self, path: &Path) -> Result { + fs::canonicalize(path).map_err(|error| error.to_string()) + } + + fn read_to_string(&self, path: &Path) -> Result { + fs::read_to_string(path).map_err(|error| error.to_string()) + } +} + +pub fn plan(inputs: &Inputs, probe: &impl PathProbe) -> Result { + validate_source_mode(&inputs.features)?; + validate_feature_implications(&inputs.features)?; + validate_targets(inputs)?; + + let cuda_architectures = validate_cuda(inputs)?; + let cutlass_dir = if inputs.features.cuda_cutlass { + Some(validate_cutlass( + inputs, + probe, + cuda_architectures.as_deref(), + )?) + } else { + None + }; + + let sanitizer = inputs.environment.sanitizer.as_deref().unwrap_or("OFF"); + if inputs.features.cuda && sanitizer != "OFF" { + return Err(ConfigError::new(format!( + "VLLM_CPP_SANITIZE={sanitizer} cannot be combined with `cuda`; use compute-sanitizer for CUDA or set VLLM_CPP_SANITIZE=OFF" + ))); + } + + let on_off = |enabled: bool| if enabled { "ON" } else { "OFF" }.to_owned(); + let mut cmake_defines = vec![ + ("VLLM_CPP_BUILD_TESTS", "OFF".to_owned()), + ("VLLM_CPP_BUILD_EXAMPLES", "OFF".to_owned()), + ("VLLM_CPP_SERVER", "OFF".to_owned()), + ("VLLM_CPP_CUDA", on_off(inputs.features.cuda)), + ("VLLM_CPP_METAL", "OFF".to_owned()), + ("VLLM_CPP_VULKAN", on_off(inputs.features.vulkan)), + ("VLLM_CPP_MLX", "OFF".to_owned()), + ("VLLM_CPP_TRITON", on_off(inputs.features.triton_aot)), + ("VLLM_CPP_TRITON_REGEN", "OFF".to_owned()), + ("VLLM_CPP_CUTLASS_FETCH", "OFF".to_owned()), + ("VLLM_CPP_SANITIZE", sanitizer.to_owned()), + ]; + if let Some(architectures) = cuda_architectures { + cmake_defines.push(("VLLM_CPP_CUDA_ARCHITECTURES", architectures)); + } + if let Some(root) = cutlass_dir { + cmake_defines.push(("VLLM_CPP_CUTLASS_DIR", root.to_string_lossy().into_owned())); + } + + let mut link_requirements = platform_link_requirements(inputs)?; + if inputs.features.cuda && !inputs.features.dynamic_link { + link_requirements.extend([ + LinkRequirement::CudaToolkit(CudaComponent::Runtime), + LinkRequirement::CudaToolkit(CudaComponent::BlasLt), + ]); + if inputs.features.triton_aot { + link_requirements.push(LinkRequirement::CudaToolkit(CudaComponent::Driver)); + } + } + + Ok(BuildPlan { + cmake_defines, + link_requirements, + }) +} + +fn validate_source_mode(features: &Features) -> Result<(), ConfigError> { + match (features.bundled, features.system) { + (true, true) => Err(ConfigError::new( + "features `bundled` and `system` are mutually exclusive; disable default features when selecting `system`", + )), + (false, false) => Err(ConfigError::new( + "enable exactly one of the `bundled` or `system` features", + )), + (false, true) if features.has_backend() => Err(ConfigError::new( + "accelerator features are bundled-only and cannot be combined with `system`; build the system library with its own backend configuration and select only `system`", + )), + _ => Ok(()), + } +} + +fn validate_feature_implications(features: &Features) -> Result<(), ConfigError> { + if features.cuda_cutlass && !features.cuda { + return Err(ConfigError::new( + "`cuda-cutlass` requires the `cuda` feature", + )); + } + if features.triton_aot && !features.cuda { + return Err(ConfigError::new("`triton-aot` requires the `cuda` feature")); + } + if features.cuda && features.vulkan { + return Err(ConfigError::new( + "`cuda` and `vulkan` cannot be combined in release 0.1; build separate backend artifacts", + )); + } + Ok(()) +} + +fn validate_targets(inputs: &Inputs) -> Result<(), ConfigError> { + if inputs.target.os != "linux" { + return Err(ConfigError::new(format!( + "linking is implemented only for Linux, not target {}", + inputs.target.triple + ))); + } + if (inputs.features.cuda || inputs.features.vulkan) + && !matches!(inputs.target.arch.as_str(), "x86_64" | "aarch64") + { + let feature = if inputs.features.cuda { + "cuda" + } else { + "vulkan" + }; + return Err(ConfigError::new(format!( + "`{feature}` supports only Linux x86_64/aarch64 targets; target {} is unsupported", + inputs.target.triple + ))); + } + Ok(()) +} + +fn validate_cuda(inputs: &Inputs) -> Result, ConfigError> { + if !inputs.features.cuda { + if inputs.environment.cuda_architectures.is_some() { + return Err(ConfigError::new( + "VLLM_CPP_CUDA_ARCHITECTURES is set but the `cuda` feature is disabled; remove it or enable `cuda`", + )); + } + return Ok(None); + } + + let architectures = inputs + .environment + .cuda_architectures + .as_deref() + .ok_or_else(|| { + ConfigError::new(format!( + "the `cuda` feature requires an exact VLLM_CPP_CUDA_ARCHITECTURES value: {}", + CUDA_ARCHITECTURES.join(", ") + )) + })?; + if !CUDA_ARCHITECTURES.contains(&architectures) { + return Err(ConfigError::new(format!( + "unsupported VLLM_CPP_CUDA_ARCHITECTURES={architectures:?}; supported values are {}", + CUDA_ARCHITECTURES.join(", ") + ))); + } + if inputs.features.triton_aot && !TRITON_ARCHITECTURES.contains(&architectures) { + return Err(ConfigError::new(format!( + "`triton-aot` requires one vendored single architecture: {}; got {architectures:?}", + TRITON_ARCHITECTURES.join(", ") + ))); + } + Ok(Some(architectures.to_owned())) +} + +fn validate_cutlass( + inputs: &Inputs, + probe: &impl PathProbe, + cuda_architectures: Option<&str>, +) -> Result { + let root = inputs.environment.cutlass_dir.as_deref().ok_or_else(|| { + ConfigError::new( + "`cuda-cutlass` requires VLLM_CPP_CUTLASS_DIR pointing to an existing CUTLASS >=4.5.0 checkout; it is never fetched", + ) + })?; + if !probe.is_dir(root) { + return Err(ConfigError::new(format!( + "VLLM_CPP_CUTLASS_DIR={} is not an existing directory", + root.display() + ))); + } + let canonical = probe.canonicalize(root).map_err(|error| { + ConfigError::new(format!( + "failed to canonicalize VLLM_CPP_CUTLASS_DIR={}: {error}", + root.display() + )) + })?; + if !canonical.is_absolute() { + return Err(ConfigError::new(format!( + "VLLM_CPP_CUTLASS_DIR must resolve to an absolute path; got {}", + canonical.display() + ))); + } + + for relative in ["include/cutlass/cutlass.h", "include/cutlass/version.h"] { + let path = canonical.join(relative); + if !probe.is_file(&path) { + return Err(ConfigError::new(format!( + "VLLM_CPP_CUTLASS_DIR must contain {relative}; missing {}", + path.display() + ))); + } + } + let tools = canonical.join("tools/util/include"); + if !probe.is_dir(&tools) { + return Err(ConfigError::new(format!( + "VLLM_CPP_CUTLASS_DIR must contain tools/util/include; missing {}", + tools.display() + ))); + } + + let version_path = canonical.join("include/cutlass/version.h"); + let contents = probe.read_to_string(&version_path).map_err(|error| { + ConfigError::new(format!( + "failed to read CUTLASS version from {}: {error}", + version_path.display() + )) + })?; + let version = parse_cutlass_version(&contents).ok_or_else(|| { + ConfigError::new(format!( + "could not parse CUTLASS_MAJOR, CUTLASS_MINOR, and CUTLASS_PATCH from {}", + version_path.display() + )) + })?; + if version < (4, 5, 0) { + return Err(ConfigError::new(format!( + "CUTLASS >=4.5.0 is required; found {}.{}.{} at {}", + version.0, + version.1, + version.2, + canonical.display() + ))); + } + if matches!(cuda_architectures, Some("103a" | "110")) { + return Err(ConfigError::new(format!( + "`cuda-cutlass` has no enabled upstream kernel for CUDA architecture {}; remove `cuda-cutlass` for this portable-kernel build", + cuda_architectures.expect("matched Some") + ))); + } + Ok(canonical) +} + +fn parse_cutlass_version(contents: &str) -> Option<(u32, u32, u32)> { + fn value(contents: &str, name: &str) -> Option { + contents.lines().find_map(|line| { + let mut words = line.split_whitespace(); + (words.next()? == "#define" && words.next()? == name) + .then(|| words.next()?.parse().ok())? + }) + } + + Some(( + value(contents, "CUTLASS_MAJOR")?, + value(contents, "CUTLASS_MINOR")?, + value(contents, "CUTLASS_PATCH")?, + )) +} + +fn platform_link_requirements(inputs: &Inputs) -> Result, ConfigError> { + if inputs.features.dynamic_link { + return Ok(Vec::new()); + } + if inputs.target.os != "linux" { + return Err(ConfigError::new(format!( + "static linking is implemented only for Linux, not {}", + inputs.target.os + ))); + } + Ok(vec![ + LinkRequirement::Library("stdc++"), + LinkRequirement::Library("pthread"), + LinkRequirement::Library("dl"), + ]) +} diff --git a/vllm-cpp-sys/src/build_support.rs b/vllm-cpp-sys/src/build_support.rs index 63e62a9..cd83911 100644 --- a/vllm-cpp-sys/src/build_support.rs +++ b/vllm-cpp-sys/src/build_support.rs @@ -65,3 +65,42 @@ pub(crate) fn find_installed_library_dir( )), } } + +pub(crate) fn cmake_cache_path(cache: &Path, key: &str) -> Result { + let contents = fs::read_to_string(cache) + .map_err(|error| format!("could not read {}: {error}", cache.display()))?; + let prefix = format!("{key}:"); + let matches: Vec<&str> = contents + .lines() + .filter_map(|line| Some(line.strip_prefix(&prefix)?.split_once('=')?.1)) + .collect(); + + let value = match matches.as_slice() { + [value] => *value, + [] => return Err(format!("{key} is absent from {}", cache.display())), + _ => { + return Err(format!( + "{key} occurs more than once in {}", + cache.display() + )); + } + }; + if value.is_empty() { + return Err(format!("{key} is empty in {}", cache.display())); + } + if value.ends_with("-NOTFOUND") { + return Err(format!( + "{key} is unresolved ({value}) in {}", + cache.display() + )); + } + + let path = PathBuf::from(value); + if !path.is_absolute() { + return Err(format!( + "{key} must be an absolute path in {}; got {value:?}", + cache.display() + )); + } + Ok(path) +} diff --git a/vllm-cpp-sys/tests/build_config.rs b/vllm-cpp-sys/tests/build_config.rs new file mode 100644 index 0000000..7c0245a --- /dev/null +++ b/vllm-cpp-sys/tests/build_config.rs @@ -0,0 +1,338 @@ +#![allow(dead_code)] + +#[path = "../src/build_config.rs"] +mod build_config; + +use std::collections::{BTreeMap, BTreeSet}; +use std::path::{Path, PathBuf}; + +use build_config::{ + plan, CudaComponent, Environment, Features, Inputs, LinkRequirement, PathProbe, Target, + ERROR_PREFIX, +}; + +#[derive(Default)] +struct MockProbe { + canonical: BTreeMap, + files: BTreeMap, + dirs: BTreeSet, +} + +impl MockProbe { + fn cutlass(version: (u32, u32, u32)) -> Self { + let mut probe = Self::default(); + probe + .canonical + .insert(PathBuf::from("cutlass"), PathBuf::from("/cutlass")); + probe.dirs.insert(PathBuf::from("cutlass")); + probe.dirs.insert(PathBuf::from("/cutlass")); + probe + .dirs + .insert(PathBuf::from("/cutlass/tools/util/include")); + probe.files.insert( + PathBuf::from("/cutlass/include/cutlass/cutlass.h"), + String::new(), + ); + probe.files.insert( + PathBuf::from("/cutlass/include/cutlass/version.h"), + format!( + "#define CUTLASS_MAJOR {}\n#define CUTLASS_MINOR {}\n#define CUTLASS_PATCH {}\n", + version.0, version.1, version.2 + ), + ); + probe + } +} + +impl PathProbe for MockProbe { + fn is_file(&self, path: &Path) -> bool { + self.files.contains_key(path) + } + + fn is_dir(&self, path: &Path) -> bool { + self.dirs.contains(path) + } + + fn canonicalize(&self, path: &Path) -> Result { + self.canonical + .get(path) + .cloned() + .or_else(|| path.is_absolute().then(|| path.to_path_buf())) + .ok_or_else(|| "not found".to_owned()) + } + + fn read_to_string(&self, path: &Path) -> Result { + self.files + .get(path) + .cloned() + .ok_or_else(|| "not found".to_owned()) + } +} + +fn linux() -> Inputs { + Inputs { + features: Features { + bundled: true, + ..Features::default() + }, + target: Target { + triple: "x86_64-unknown-linux-gnu".to_owned(), + os: "linux".to_owned(), + arch: "x86_64".to_owned(), + }, + environment: Environment::default(), + } +} + +fn error(inputs: &Inputs, probe: &MockProbe) -> String { + plan(inputs, probe).unwrap_err().to_string() +} + +#[test] +fn cpu_and_vulkan_plans_are_deterministic() { + let cpu = plan(&linux(), &MockProbe::default()).unwrap(); + assert!(cpu + .cmake_defines + .contains(&("VLLM_CPP_CUDA", "OFF".to_owned()))); + assert!(cpu + .cmake_defines + .contains(&("VLLM_CPP_VULKAN", "OFF".to_owned()))); + assert!(!cpu + .cmake_defines + .iter() + .any(|(key, _)| *key == "VLLM_CPP_CUDA_ARCHITECTURES")); + assert_eq!( + cpu.link_requirements, + vec![ + LinkRequirement::Library("stdc++"), + LinkRequirement::Library("pthread"), + LinkRequirement::Library("dl"), + ] + ); + + let mut vulkan = linux(); + vulkan.features.vulkan = true; + let first = plan(&vulkan, &MockProbe::default()).unwrap(); + let second = plan(&vulkan, &MockProbe::default()).unwrap(); + assert_eq!(first, second); + assert!(first + .cmake_defines + .contains(&("VLLM_CPP_VULKAN", "ON".to_owned()))); +} + +#[test] +fn accepts_exact_cuda_architecture_spellings() { + for architecture in [ + "80", + "86", + "87", + "89", + "90a", + "100a", + "103a", + "110", + "120a", + "121a", + "120a;121a", + ] { + let mut inputs = linux(); + inputs.features.cuda = true; + inputs.environment.cuda_architectures = Some(architecture.to_owned()); + let plan = plan(&inputs, &MockProbe::default()).unwrap(); + assert!(plan + .cmake_defines + .contains(&("VLLM_CPP_CUDA_ARCHITECTURES", architecture.to_owned()))); + } + + for invalid in ["", "75", "90", "121", "121a;120a", "80;86", " 80"] { + let mut inputs = linux(); + inputs.features.cuda = true; + inputs.environment.cuda_architectures = Some(invalid.to_owned()); + assert!(error(&inputs, &MockProbe::default()).contains("unsupported")); + } +} + +#[test] +fn cuda_requires_arch_and_supported_target() { + let mut inputs = linux(); + inputs.features.cuda = true; + assert!(error(&inputs, &MockProbe::default()).contains("requires an exact")); + + inputs.environment.cuda_architectures = Some("80".to_owned()); + inputs.target = Target { + triple: "x86_64-pc-windows-msvc".to_owned(), + os: "windows".to_owned(), + arch: "x86_64".to_owned(), + }; + assert!(error(&inputs, &MockProbe::default()).contains("only for Linux")); +} + +#[test] +fn rejects_source_mode_and_backend_conflicts() { + let mut both = linux(); + both.features.system = true; + assert!(error(&both, &MockProbe::default()).contains("mutually exclusive")); + + let mut neither = linux(); + neither.features.bundled = false; + assert!(error(&neither, &MockProbe::default()).contains("exactly one")); + + let mut system_cuda = linux(); + system_cuda.features.bundled = false; + system_cuda.features.system = true; + system_cuda.features.cuda = true; + assert!(error(&system_cuda, &MockProbe::default()).contains("bundled-only")); + + let mut cuda_vulkan = linux(); + cuda_vulkan.features.cuda = true; + cuda_vulkan.features.vulkan = true; + assert!(error(&cuda_vulkan, &MockProbe::default()).contains("cannot be combined")); +} + +#[test] +fn rejects_broken_feature_implications() { + let mut cutlass = linux(); + cutlass.features.cuda_cutlass = true; + assert!(error(&cutlass, &MockProbe::default()).contains("requires the `cuda`")); + + let mut triton = linux(); + triton.features.triton_aot = true; + assert!(error(&triton, &MockProbe::default()).contains("requires the `cuda`")); +} + +#[test] +fn validates_cutlass_tree_version_architecture_and_canonical_root() { + let mut inputs = linux(); + inputs.features.cuda = true; + inputs.features.cuda_cutlass = true; + inputs.environment.cuda_architectures = Some("80".to_owned()); + assert!(error(&inputs, &MockProbe::default()).contains("VLLM_CPP_CUTLASS_DIR")); + + inputs.environment.cutlass_dir = Some(PathBuf::from("cutlass")); + assert!(error(&inputs, &MockProbe::default()).contains("not an existing directory")); + assert!(error(&inputs, &MockProbe::cutlass((4, 4, 2))).contains(">=4.5.0")); + + let plan = plan(&inputs, &MockProbe::cutlass((4, 5, 0))).unwrap(); + assert!(plan + .cmake_defines + .contains(&("VLLM_CPP_CUTLASS_DIR", "/cutlass".to_owned()))); + + for unsupported in ["103a", "110"] { + inputs.environment.cuda_architectures = Some(unsupported.to_owned()); + assert!(error(&inputs, &MockProbe::cutlass((4, 6, 1))).contains("no enabled")); + } +} + +#[test] +fn cutlass_requires_all_paths_and_parseable_version() { + let mut inputs = linux(); + inputs.features.cuda = true; + inputs.features.cuda_cutlass = true; + inputs.environment.cuda_architectures = Some("121a".to_owned()); + inputs.environment.cutlass_dir = Some(PathBuf::from("cutlass")); + + let mut probe = MockProbe::cutlass((4, 5, 0)); + probe + .files + .remove(Path::new("/cutlass/include/cutlass/cutlass.h")); + assert!(error(&inputs, &probe).contains("cutlass.h")); + + let mut probe = MockProbe::cutlass((4, 5, 0)); + probe.dirs.remove(Path::new("/cutlass/tools/util/include")); + assert!(error(&inputs, &probe).contains("tools/util/include")); + + let mut probe = MockProbe::cutlass((4, 5, 0)); + probe.files.insert( + PathBuf::from("/cutlass/include/cutlass/version.h"), + "not a version".to_owned(), + ); + assert!(error(&inputs, &probe).contains("could not parse")); +} + +#[test] +fn static_cuda_links_exact_components_while_dynamic_relies_on_dt_needed() { + let mut inputs = linux(); + inputs.features.cuda = true; + inputs.features.triton_aot = true; + inputs.environment.cuda_architectures = Some("80".to_owned()); + + let static_plan = plan(&inputs, &MockProbe::default()).unwrap(); + assert_eq!( + static_plan.link_requirements, + vec![ + LinkRequirement::Library("stdc++"), + LinkRequirement::Library("pthread"), + LinkRequirement::Library("dl"), + LinkRequirement::CudaToolkit(CudaComponent::Runtime), + LinkRequirement::CudaToolkit(CudaComponent::BlasLt), + LinkRequirement::CudaToolkit(CudaComponent::Driver), + ] + ); + + inputs.features.dynamic_link = true; + let dynamic = plan(&inputs, &MockProbe::default()).unwrap(); + assert!(dynamic.link_requirements.is_empty()); +} + +#[test] +fn cuda_cache_keys_match_find_cudatoolkit() { + assert_eq!(CudaComponent::Runtime.cmake_cache_key(), "CUDA_CUDART"); + assert_eq!( + CudaComponent::BlasLt.cmake_cache_key(), + "CUDA_cublasLt_LIBRARY" + ); + assert_eq!( + CudaComponent::Driver.cmake_cache_key(), + "CUDA_cuda_driver_LIBRARY" + ); + assert_eq!(CudaComponent::Runtime.cargo_library(), "cudart"); + assert_eq!(CudaComponent::BlasLt.cargo_library(), "cublasLt"); + assert_eq!(CudaComponent::Driver.cargo_library(), "cuda"); +} + +#[test] +fn triton_accepts_only_vendored_single_architectures() { + for architecture in ["80", "86", "89", "90a", "100a", "121a"] { + let mut inputs = linux(); + inputs.features.cuda = true; + inputs.features.triton_aot = true; + inputs.environment.cuda_architectures = Some(architecture.to_owned()); + let plan = plan(&inputs, &MockProbe::default()).unwrap(); + assert!(plan + .cmake_defines + .contains(&("VLLM_CPP_TRITON", "ON".to_owned()))); + assert!(plan + .cmake_defines + .contains(&("VLLM_CPP_TRITON_REGEN", "OFF".to_owned()))); + } + + for invalid in ["87", "103a", "110", "120a", "120a;121a"] { + let mut inputs = linux(); + inputs.features.cuda = true; + inputs.features.triton_aot = true; + inputs.environment.cuda_architectures = Some(invalid.to_owned()); + assert!(error(&inputs, &MockProbe::default()).contains("vendored single")); + } +} + +#[test] +fn rejects_cuda_sanitizers_and_unsupported_vulkan_targets() { + let mut inputs = linux(); + inputs.features.cuda = true; + inputs.environment.cuda_architectures = Some("80".to_owned()); + inputs.environment.sanitizer = Some("address,undefined".to_owned()); + assert!(error(&inputs, &MockProbe::default()).contains("compute-sanitizer")); + + let mut vulkan = linux(); + vulkan.features.vulkan = true; + vulkan.target.arch = "riscv64".to_owned(); + vulkan.target.triple = "riscv64gc-unknown-linux-gnu".to_owned(); + assert!(error(&vulkan, &MockProbe::default()).contains("Linux x86_64/aarch64")); +} + +#[test] +fn every_error_has_the_actionable_prefix() { + let mut inputs = linux(); + inputs.features.cuda = true; + assert!(error(&inputs, &MockProbe::default()).starts_with(ERROR_PREFIX)); +} diff --git a/vllm-cpp-sys/tests/build_support.rs b/vllm-cpp-sys/tests/build_support.rs index 11b6649..c56682e 100644 --- a/vllm-cpp-sys/tests/build_support.rs +++ b/vllm-cpp-sys/tests/build_support.rs @@ -6,7 +6,9 @@ use std::fs; use std::path::{Path, PathBuf}; use std::time::{SystemTime, UNIX_EPOCH}; -use build_support::{find_installed_library_dir, require_library_file, shared_library_name}; +use build_support::{ + cmake_cache_path, find_installed_library_dir, require_library_file, shared_library_name, +}; struct TempDir { path: PathBuf, @@ -37,11 +39,15 @@ impl Drop for TempDir { } } -fn write_file(path: &Path) { +fn write_contents(path: &Path, contents: &[u8]) { if let Some(parent) = path.parent() { fs::create_dir_all(parent).expect("failed to create parent directory"); } - fs::write(path, b"archive").expect("failed to write test artifact"); + fs::write(path, contents).expect("failed to write test artifact"); +} + +fn write_file(path: &Path) { + write_contents(path, b"archive"); } #[test] @@ -130,3 +136,47 @@ fn require_library_file_reports_the_expected_path() { .to_string() )); } + +#[test] +fn cmake_cache_path_requires_one_nonempty_absolute_value() { + let temp = TempDir::new("cmake-cache-errors"); + let cache = temp.path().join("CMakeCache.txt"); + + write_contents(&cache, b"OTHER:FILEPATH=/tmp/libother.so\n"); + let error = cmake_cache_path(&cache, "CUDA_CUDART").expect_err("key should be absent"); + assert!(error.contains("CUDA_CUDART is absent")); + + write_contents(&cache, b"CUDA_CUDART:FILEPATH=\n"); + let error = cmake_cache_path(&cache, "CUDA_CUDART").expect_err("value should be empty"); + assert!(error.contains("CUDA_CUDART is empty")); + + write_contents(&cache, b"CUDA_CUDART:FILEPATH=CUDA_CUDART-NOTFOUND\n"); + let error = cmake_cache_path(&cache, "CUDA_CUDART").expect_err("value should be unresolved"); + assert!(error.contains("CUDA_CUDART is unresolved")); + + write_contents(&cache, b"CUDA_CUDART:FILEPATH=libcudart.so\n"); + let error = cmake_cache_path(&cache, "CUDA_CUDART").expect_err("value should be relative"); + assert!(error.contains("must be an absolute path")); + + write_contents( + &cache, + b"CUDA_CUDART:FILEPATH=/cuda/lib/libcudart.so\nCUDA_CUDART:STRING=/other/libcudart.so\n", + ); + let error = cmake_cache_path(&cache, "CUDA_CUDART").expect_err("key should be duplicated"); + assert!(error.contains("occurs more than once")); +} + +#[test] +fn cmake_cache_path_returns_the_exact_absolute_value() { + let temp = TempDir::new("cmake-cache-valid"); + let cache = temp.path().join("CMakeCache.txt"); + write_contents( + &cache, + b"CUDA_cublasLt_LIBRARY:FILEPATH=/nix/store/toolkit/lib/libcublasLt.so\n", + ); + + assert_eq!( + cmake_cache_path(&cache, "CUDA_cublasLt_LIBRARY").unwrap(), + PathBuf::from("/nix/store/toolkit/lib/libcublasLt.so") + ); +} diff --git a/vllm-cpp/Cargo.toml b/vllm-cpp/Cargo.toml index b411250..6d1571a 100644 --- a/vllm-cpp/Cargo.toml +++ b/vllm-cpp/Cargo.toml @@ -12,6 +12,10 @@ default = ["bundled"] bundled = ["vllm-cpp-sys/bundled"] system = ["vllm-cpp-sys/system"] dynamic-link = ["vllm-cpp-sys/dynamic-link"] +cuda = ["vllm-cpp-sys/cuda"] +cuda-cutlass = ["cuda", "vllm-cpp-sys/cuda-cutlass"] +triton-aot = ["cuda", "vllm-cpp-sys/triton-aot"] +vulkan = ["vllm-cpp-sys/vulkan"] serde = ["dep:serde_json"] [dependencies] diff --git a/vllm-cpp/examples/README.md b/vllm-cpp/examples/README.md new file mode 100644 index 0000000..81e0e19 --- /dev/null +++ b/vllm-cpp/examples/README.md @@ -0,0 +1,102 @@ +# examples + +these examples exercise the safe `vllm-cpp` api with fixed prompts and settings: + +| example | behavior | +|---|---| +| [`complete`](complete.rs) | runs one blocking text completion | +| [`stream`](stream.rs) | prints one completion as token deltas arrive | +| [`concurrent`](concurrent.rs) | submits two asynchronous streaming requests and waits for both | +| [`chat`](chat.rs) | sends a fixed raw-json chat request; the optional `serde` feature is not required | +| [`structured`](structured.rs) | constrains one completion to the choice `red` or `blue` | + +each example reads the first positional argument as a model directory and implements no additional options. a usable directory must contain the runtime files `model.safetensors`, `config.json`, `tokenizer.json`, and `tokenizer_config.json`; the pinned [test model fixture and layout](https://github.com/querymt/vllm-cpp-rs#test-model-and-sanitizers) is the known-good reference. model compatibility depends on the native engine, so an arbitrary model directory is not guaranteed to work. + +## ordinary linux + +nix and nixos are not required. for the default bundled cpu build, install: + +- rust and cargo +- cmake 3.24 or newer +- ninja or another cmake build tool +- a c11 compiler, a c++20 compiler, a linker, and a c++ standard library +- git, so the pinned vllm.cpp submodule can be initialized + +`just`, `jq`, bindgen, and libclang are maintainer tools and are not required to run these examples. from the workspace root, initialize the native source once: + +```console +git submodule update --init --recursive +``` + +the commands below select ninja explicitly with `CMAKE_GENERATOR=Ninja`; merely installing ninja does not configure cmake to use it. if you choose another cmake generator and build tool, install them and set or otherwise configure `CMAKE_GENERATOR` accordingly. + +the common command shape is: + +```console +CMAKE_GENERATOR=Ninja \ + cargo run --locked --release -p vllm-cpp --features bundled --example EXAMPLE -- /path/to/model +``` + +`bundled` is the default feature; it is shown explicitly here to identify the cpu backend. run any of the five examples with: + +```console +CMAKE_GENERATOR=Ninja \ + cargo run --locked --release -p vllm-cpp --features bundled --example complete -- /path/to/model +CMAKE_GENERATOR=Ninja \ + cargo run --locked --release -p vllm-cpp --features bundled --example stream -- /path/to/model +CMAKE_GENERATOR=Ninja \ + cargo run --locked --release -p vllm-cpp --features bundled --example concurrent -- /path/to/model +CMAKE_GENERATOR=Ninja \ + cargo run --locked --release -p vllm-cpp --features bundled --example chat -- /path/to/model +CMAKE_GENERATOR=Ninja \ + cargo run --locked --release -p vllm-cpp --features bundled --example structured -- /path/to/model +``` + +## optional nix shell + +nix is optional and works on supported linux installations with nix; it does not require nixos. the default development shell supplies the cpu build dependencies. from the workspace root, run: + +```console +CMAKE_GENERATOR=Ninja \ + nix develop -c cargo run --locked --release -p vllm-cpp --features bundled --example complete -- /path/to/model +``` + +replace `complete` with any other example name from the table. + +## experimental cuda + +plain `cuda` is the baseline accelerator feature. read the root [experimental backend build details](https://github.com/querymt/vllm-cpp-rs#experimental-backend-builds) before using it: cuda is a bundled linux experimental/build-only integration surface, and successful compilation does not guarantee model inference or backend correctness. a non-nix build needs a compatible cuda toolkit and driver installation in addition to the ordinary linux prerequisites. + +set `VLLM_CPP_CUDA_ARCHITECTURES` to an architecture supported by this crate and the target gpu, and use a fresh `CARGO_TARGET_DIR` for the backend/link combination. for example, the tested RTX 5080 setup uses `120a`; do not use that value for unrelated hardware: + +```console +arch=120a +CMAKE_GENERATOR=Ninja \ + VLLM_CPP_CUDA_ARCHITECTURES="$arch" \ + CARGO_TARGET_DIR="$PWD/target/cuda-examples" \ + cargo run --locked --release -p vllm-cpp --features cuda --example complete -- /path/to/model +``` + +with nix installed, the cuda shell supplies the pinned toolkit dependencies but still requires an explicit target architecture: + +```console +arch=120a +CMAKE_GENERATOR=Ninja \ + VLLM_CPP_CUDA_ARCHITECTURES="$arch" \ + CARGO_TARGET_DIR="$PWD/target/cuda-examples-nix" \ + nix develop .#cuda -c cargo run --locked --release -p vllm-cpp --features cuda --example complete -- /path/to/model +``` + +other accelerator features have stricter limits: + +- `cuda-cutlass` is an optional cuda variant; follow the [exact external cutlass prerequisites and known blockers](https://github.com/querymt/vllm-cpp-rs#experimental-backend-builds) before selecting it. +- `triton-aot` requires a target architecture with matching checked-in artifacts; `120a` is not supported by those artifacts. +- `vulkan` is currently for backend build/testing work. its model attention path is absent, so it cannot run these full-model examples. + +## troubleshooting + +- if the native source or cmake inputs are missing, rerun `git submodule update --init --recursive`. +- the first bundled build compiles the native c++ library and can take substantially longer than later runs. +- if model loading fails, verify the directory argument and its model, configuration, and tokenizer files, then confirm that vllm.cpp supports the model. +- for `dynamic-link` or `system` builds, follow the root [link mode and loader-path requirements](https://github.com/querymt/vllm-cpp-rs#link-modes); cargo does not deploy `libvllm.so` or configure its runtime search path. +- for accelerator configuration errors, use a fresh target directory and consult the root [experimental backend build details](https://github.com/querymt/vllm-cpp-rs#experimental-backend-builds).