Trait-based test lanes: filterable inner loop, and PR CI on a stock runner - #156
Merged
Conversation
The suite mixes four populations in one assembly: self-contained correctness tests, benchmarks, CUDA/MLX-gated tests, and tests needing real GGUF weights — the gated ones silently pass when their prerequisite is absent, and the benchmarks flake under full-suite load. Class-level (method-level for the three mixed classes) traits make each population selectable: Category=Bench 39 benchmark tests across 11 classes Requires=Cuda 164 tests needing a CUDA device Requires=Mlx 94 tests needing the MLX backend Requires=Models 89 tests needing real weights (TS_TEST_MODEL_DIR etc.) Membership was determined from each class's actual gate (availability early-returns, model-dir env lookups), not names: e.g. ConfigFileArgsTests and ServerOptionsBuilderTests mention .gguf but are pure-unit and stay untagged; SafetensorsReaderTests, WanDirectOpsTests, and Gemma4PromptRenderReproTests are mixed and tagged per method. ToolFunctionParserBenchmark asserts correctness rather than timing, but its measurement iterations are pure wall clock for the inner loop and its numbers only mean something on a quiet box, so it rides the Bench lane with the other harnesses. Verified on a machine without CUDA/MLX/model weights: unfiltered run unchanged (1474 passed, 3m29s); Category!=Bench = 1435 passed in 8s (the benches carry ~95% of the suite's wall clock); the inner-loop lane (no bench/models/cuda/mlx) = 1110 passed in 5s. Lane filters documented in TestAssemblyConfig.cs.
DEVELOPMENT.md (and the zh-cn twin) gets a Test lanes section under Unit tests showing the three filter invocations: the inner loop (environment-independent correctness), full correctness (everything but benchmarks), and the bench lane to run deliberately on quiet hardware. pr-unit-tests.yml runs the inner-loop lane on ubuntu-latest for every PR: no GPU, no model downloads, no secrets — a deterministic test signal that doesn't depend on the self-hosted CUDA runner being online. The managed build triggers the CPU-only native GGML build itself; Vulkan is pinned off since tests default to the CPU backend. The engine-comparison benchmark workflow is untouched.
Owner
|
@craig-b Thanks for this update. It really makes the entire test process be better managed. Merged it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements #153 — plain xUnit traits that make the existing test suite filterable into lanes, plus a minimal PR workflow that runs the environment-independent lane on a stock GitHub-hosted runner.
What's in the diff
Trait tagging (43 files, additive only — no test moves, no behavior change):
[Trait]attributes on the existing classes, method-level for the three mixed classes. Membership was determined from each class's actual gate (availability early-returns, model-dir env lookups), not from names — e.g.ConfigFileArgsTestsmentions.ggufbut is pure-unit and stays untagged.Category=BenchRequires=CudaRequires=MlxRequires=ModelsUntagged = self-contained correctness tests that run anywhere. The taxonomy and lane filters are documented in
InferenceWeb.Tests/TestAssemblyConfig.cs.Lane documentation: a "Test lanes" section in DEVELOPMENT.md (and the zh-cn twin) with the three filter invocations — inner loop, full correctness, bench-only.
PR CI (
.github/workflows/pr-unit-tests.yml): runs the inner-loop filter onubuntu-latestfor every PR — no GPU, no model downloads, no secrets — so PRs get a deterministic test signal even when the self-hosted CUDA runner is offline. The managed build already triggers the CPU-only native GGML build by itself; Vulkan is pinned off since tests default to the CPU backend. The engine-comparison benchmark workflow (test-matrix.yml) is untouched.Verified on a Linux box without CUDA/MLX/model weights, rebased on current main:
dotnet test--filter "Category!=Bench"One thing this PR deliberately does not do: CUDA/MLX/model-gated tests still silently pass when their prerequisite is missing. Turning those into visible xUnit Skipped counts needs custom fact attributes; happy to follow up separately if you want that.