Report environment-gated tests as skipped instead of silently passing - #157
Open
craig-b wants to merge 2 commits into
Open
Report environment-gated tests as skipped instead of silently passing#157craig-b wants to merge 2 commits into
craig-b wants to merge 2 commits into
Conversation
…tributes CudaFact/CudaTheory, MlxFact/MlxTheory, and ModelFact/ModelTheory set Skip at discovery when the prerequisite (CUDA device, MLX backend, or GGUF weights via an env var + optional filename substring) is missing, and apply the matching Requires trait through a discoverer, replacing the per-class [Trait] lines and the silent early-return idiom: a bare box now reports these tests as Skipped instead of Passed. Lane filters are unchanged. A skipped [Theory] reports once rather than per data row. Classes whose prerequisite does not reduce to one env probe (multi-var gates, per-method model choice, hardcoded fixture paths) keep explicit traits and in-body gating: EngineParallelInferenceTests, Gemma4AudioEncoderTests, Gemma4MtpTests, ParallelThroughputBench, Qwen36MtpTests, Qwen3BatchedForwardTests, SafetensorsReaderTests.
BannedApiAnalyzers with RS0030 as error: CudaBackend.IsAvailable() and MlxBackend.IsAvailable() are banned in InferenceWeb.Tests, so a new hardware-gated test cannot silently hand-roll the old early-return gate — the gated attributes are the only way through. The probe cache in GatedFacts.cs and the two probe-under-test methods in MlxBackendTests are pragma-exempted. Lane docs updated in both DEVELOPMENT files.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #156 / #153. The traits work left one problem in place, called out in the issue: a test whose prerequisite is missing (no CUDA device, no MLX backend, no GGUF weights) early-returns and counts as Passed, so
dotnet teston a bare machine reads "all green" while a third of the suite never executed.This PR makes those tests report as Skipped:
InferenceWeb.Tests/GatedFacts.csadds[CudaFact]/[CudaTheory],[MlxFact]/[MlxTheory], and[ModelFact("ENV_VAR", "gguf-substring")]/[ModelTheory(...)]. Each setsSkipwith a reason at discovery when its prerequisite is missing, and applies the matchingRequirestrait automatically through a trait discoverer — so the lane filters from Trait-based test lanes: filterable inner loop, and PR CI on a stock runner #156 work exactly as before, and gated tests no longer need a separate[Trait]line. Plain xunit v2, no custom runners or test-case discoverers.Requirestrait and the in-bodyif (!...IsAvailable()) return;/ "env not set; skipping" blocks (net −322 lines). Loader helpers are untouched; their null-checks remain as a backstop for load errors.TS_GMTP_*, per-method model selection inParallelThroughputBench, hardcoded fixture paths inSafetensorsReaderTests). They behave exactly as before.Microsoft.CodeAnalysis.BannedApiAnalyzersbansCudaBackend.IsAvailable()/MlxBackend.IsAvailable()in test code (RS0030 as error, message points at the attributes). The probe cache inGatedFacts.csand the two probe-under-test methods inMlxBackendTestsare pragma-exempted.DEVELOPMENT.mdandDEVELOPMENT_zh-cn.md.Behavior change to be aware of: on a machine missing prerequisites, a green run now reads e.g. "1195 passed, 190 skipped" instead of "1385 passed". Also, a skipped
[Theory]reports once rather than once per data row, so skip counts are per-method; on a machine that has the hardware, row-level counts are unchanged.Verified locally (Linux, AMD GPU — no CUDA, no MLX, no test weights): inner-loop lane 1158 passed / 0 skipped in 8s, identical selection to before;
Category!=Bench1195 passed / 190 skipped / 0 failed; the RS0030 ban confirmed to fail the build on a direct probe call. The #156 PR CI lane is unaffected (it filters these tests out either way).