feat: Add native Hugging Face PyTorch eval runtime

docs/commands/eval.md

@@ -1,6 +1,6 @@
 # winml eval
 
-> Evaluate ONNX model accuracy on a standard dataset.
+> Evaluate ONNX or native Hugging Face PyTorch model accuracy on a standard dataset.
 
 ## When to use this
 
@@ -26,6 +26,7 @@ $ winml eval [options]
 | `--input-specs` | | `PATH` | — | JSON input tensor specs to merge into the Hugging Face export config. Symbolic string dimensions infer dynamic axes. **Ignored for pre-built `.onnx` inputs**. |
 | `--export-config` | | `PATH` | — | JSON ONNX export config overrides (opset version, constant folding, etc.) to merge into the Hugging Face export config. **Ignored for pre-built `.onnx` inputs**. |
 | `--dynamic-axes` | | `PATH` | — | JSON dynamic axes mapping for Hugging Face ONNX export, for example `{"input_ids": {"0": "batch", "1": "sequence"}}`. **Ignored for pre-built `.onnx` inputs**. |
+| `--runtime` | | `winml\|pytorch` | `winml` | Evaluation runtime. `winml` exports Hugging Face checkpoints to ONNX; `pytorch` evaluates the original checkpoint and supports `auto`, `cpu`, or CUDA-backed `gpu` devices. |
 | `--dataset` | | `TEXT` | task default | HuggingFace dataset path (e.g., `imagenet-1k`, `nyu-mll/glue`). If omitted, a default dataset is selected based on the task. |
 | `--dataset-name` | | `TEXT` | — | Dataset configuration name for multi-config datasets. |
 | `--dataset-revision` | | `TEXT` | — | Git revision (branch, tag, or commit) of the dataset to load. Use `refs/convert/parquet` for HF datasets that are only served via the parquet mirror. |
@@ -45,7 +46,9 @@ $ winml eval [options]
 
 ## How it works
 
-`winml eval` loads the model and runs the evaluation pipeline via the internal `evaluate` function (supporting both HuggingFace IDs and local ONNX files), then pulls the requested number of samples from a HuggingFace dataset. Each sample is preprocessed using the tokenizer or image processor associated with the model ID, passed through the ONNX Runtime session, and the output is compared against the ground-truth label. Aggregated metrics (accuracy, F1, etc.) are printed to the console and optionally written to a JSON file. When `-m` is an ONNX file, `--model-id` must be provided so the command knows which preprocessor and label vocabulary to use.
+`winml eval` loads the model and runs the evaluation pipeline via the internal `evaluate` function, then pulls the requested number of samples from a HuggingFace dataset. By default, Hugging Face model IDs and local checkpoints use the `winml` runtime: they are exported to ONNX and evaluated through WinML. With `--runtime pytorch`, the checkpoint-declared PyTorch class and stored dtype are preserved and the same dataset preprocessing, Hugging Face pipeline, task evaluator, and metrics run directly against that model. PyTorch `auto` selects CUDA when available and otherwise CPU; `gpu` requires CUDA. The JSON report identifies the effective runtime as `winml` or `pytorch`.
+
+Pre-built ONNX files, composite `role=path` models, GenAI bundles, text-generation evaluation, compare mode, references, and tensor input archives remain on their existing WinML paths. `--runtime pytorch` rejects those forms, along with ONNX build, export, EP, precision, quantization, optimization, analysis, and cache-related options.
 
 ## Examples
 
@@ -55,6 +58,12 @@ Evaluate a HuggingFace model using the task-default dataset:
 $ winml eval -m microsoft/resnet-50
 ```
 
+Evaluate the original Hugging Face PyTorch checkpoint on CPU without ONNX export:
+
+```bash
+$ winml eval -m microsoft/resnet-50 --runtime pytorch --device cpu
+```
+
 ```text
 Task:     image-classification
 Dataset:  timm/mini-imagenet (test, 100 samples)
@@ -163,6 +172,7 @@ model-build pipeline from the runtime compilation cache.
 - **`--shuffle` is on by default.** The random 100-sample slice changes between runs unless you pass `--no-shuffle`. Use `--no-shuffle` when comparing two model variants to ensure they see identical samples.
 - **`--streaming` skips the local cache.** Streaming mode avoids downloading the full split but prevents random shuffling on large datasets. For reproducible evaluation, download the split once and omit `--streaming`.
 - **Export overrides only apply when eval builds from a HuggingFace ID.** `--shape-config`, `--input-specs`, `--export-config`, and `--dynamic-axes` shape the ONNX export that `eval` generates when `-m` is a HuggingFace model ID. When `-m` is a pre-built `.onnx` file, there is no export step, so these flags are ignored and the command prints a warning.
+- **The PyTorch runtime accepts only Hugging Face checkpoints.** `--runtime pytorch` cannot be combined with ONNX files, composite models, GenAI bundles, compare/reference/input-data modes, EP selection, or ONNX build/export controls. Use `--device cpu`, `--device gpu` with CUDA, or `--device auto`.
 - **Column names vary across datasets.** If the evaluator raises a missing-column error, run `winml eval --schema --task <task>` to inspect the expected schema and use `--column` to remap dataset field names to the expected names.
 
 ## See also

src/winml/modelkit/commands/eval.py

@@ -22,7 +22,7 @@
 
 
 if TYPE_CHECKING:
-    from ..eval import EvalResult, WinMLEvaluationConfig
+    from ..eval import EvalResult, EvalRuntime, WinMLEvaluationConfig
     from ..utils.constants import EPNameOrAlias
 
 
@@ -115,6 +115,14 @@
         "Ignored for pre-built ONNX inputs."
     ),
 )
+@click.option(
+    "--runtime",
+    type=click.Choice(["winml", "pytorch"]),
+    default="winml",
+    show_default=True,
+    help="Evaluation runtime. 'winml' exports Hugging Face checkpoints to ONNX; "
+    "'pytorch' evaluates the original checkpoint.",
+)
 @click.option(
     "--samples",
     type=int,
@@ -166,7 +174,9 @@
     default=None,
     help="Path to a Python script that builds the evaluation dataset.",
 )
-@cli_utils.trust_remote_code_option(optional_message="Required when --dataset-script is used.")
+@cli_utils.trust_remote_code_option(
+    optional_message="Used for native Hugging Face loading and required with --dataset-script."
+)
 @cli_utils.allow_unsupported_nodes_option()
 @click.option(
     "--schema",
@@ -235,6 +245,7 @@ def eval(
     input_specs: Path | None,
     export_config: Path | None,
     dynamic_axes: Path | None,
+    runtime: EvalRuntime,
     ep: EPNameOrAlias | None,
     samples: int,
     split: str,
@@ -302,6 +313,13 @@ def eval(
     # ── 1. Build config: defaults ← config file ← CLI ──
     cfg, config_fields = _build_eval_config(ctx, config_file, column, label_mapping_path)
 
+    if cfg.runtime not in ("winml", "pytorch"):
+        raise click.UsageError(
+            f"Invalid eval runtime {cfg.runtime!r}; expected 'winml' or 'pytorch'."
+        )
+    if cfg.runtime == "pytorch":
+        _validate_pytorch_runtime_options(ctx, cfg, config_fields)
+
     if cfg.input_data is not None and cfg.mode != "compare":
         raise click.UsageError("--input-data is only valid with --mode compare.")
 
@@ -310,12 +328,25 @@ def eval(
 
     # ── 2. Resolve in place ──
     _resolve_model(cfg, model, model_id, allow_missing_model_id=cfg.reference_path is not None)
+    if cfg.runtime == "pytorch" and cfg.model_path is not None:
+        raise click.UsageError(
+            "--runtime pytorch requires a Hugging Face model ID or local Hugging Face "
+            "checkpoint; ONNX files, composite role=path models, and GenAI bundles "
+            "are not supported."
+        )
+    if cfg.runtime == "pytorch":
+        from ..eval.evaluate import _validate_pytorch_runtime_config
+
+        try:
+            _validate_pytorch_runtime_config(cfg)
+        except ValueError as error:
+            raise click.UsageError(str(error)) from error
     _resolve_reference(cfg)
     _apply_export_overrides(cfg, shape_config_path, input_specs, export_config, dynamic_axes)
     _resolve_device(cfg)
     _resolve_genai_ep(ctx, cfg)
     _resolve_label_mapping(cfg)
-    _run_dataset_script(cfg, trust_remote_code)
+    _run_dataset_script(cfg, cfg.trust_remote_code)
 
     # Refuse to clobber an existing report unless the user opted in — fail fast
     # before the (expensive) evaluation runs.
@@ -383,7 +414,12 @@ def _build_eval_config(
 
     # ── Config file layer (only explicitly-present keys) ──
     if config_file is not None:
-        _, raw = cli_utils.load_build_config(config_file)
+        from ..eval.config import _UnsupportedEvalRuntimeFieldError
+
+        try:
+            _, raw = cli_utils.load_build_config(config_file)
+        except _UnsupportedEvalRuntimeFieldError as error:
+            raise click.UsageError(str(error)) from error
 
         # Loader task as lowest-priority fallback
         loader_section = raw.get("loader") or {}
@@ -398,6 +434,14 @@ def _build_eval_config(
         # Eval section overrides loader/compile fallbacks
         eval_data = raw.get("eval")
         if eval_data:
+            eval_data = dict(eval_data)
+            legacy_runtime_fields = {"backend", "export_model"}.intersection(eval_data)
+            if legacy_runtime_fields:
+                fields = ", ".join(sorted(legacy_runtime_fields))
+                raise click.UsageError(
+                    f"Unsupported eval runtime field(s): {fields}. Use 'runtime' "
+                    "with 'winml' or 'pytorch' instead."
+                )
             config_fields.update(eval_data)
             cfg = merge_config(cfg, eval_data)
 
@@ -442,6 +486,8 @@ def _model_build_bypass(cfg: WinMLEvaluationConfig) -> _ModelBuildBypass | None:
     from ..eval.evaluate import _ModelLoaderKind, _select_model_loader
 
     loader = _select_model_loader(cfg)
+    if loader is _ModelLoaderKind.PYTORCH:
+        return _ModelBuildBypass("PyTorch runtime evaluation")
     if loader is _ModelLoaderKind.GENAI:
         return _ModelBuildBypass(
             reason="GenAI bundles",
@@ -512,6 +558,71 @@ def _resolve_model_loader_task(cfg: WinMLEvaluationConfig) -> None:
     cfg.task = _infer_task(cfg)
 
 
+_PYTORCH_RUNTIME_INCOMPATIBLE_OPTIONS: dict[str, str] = {
+    "mode": "--mode",
+    "input_data": "--input-data",
+    "reference": "--reference",
+    "ep": "--ep",
+    "precision": "--precision",
+    "quant": "--quant/--no-quant",
+    "optimize": "--optimize/--no-optimize",
+    "analyze": "--analyze/--no-analyze",
+    "max_optim_iterations": "--max-optim-iterations",
+    "shape_config_path": "--shape-config",
+    "input_specs": "--input-specs",
+    "export_config": "--export-config",
+    "dynamic_axes": "--dynamic-axes",
+    "allow_unsupported_nodes": "--allow-unsupported-nodes",
+    "skip_build": "--skip-build/--no-skip-build",
+    "use_cache": "--use-cache/--no-use-cache",
+    "rebuild": "--rebuild/--no-rebuild",
+}
+
+_PYTORCH_RUNTIME_INCOMPATIBLE_CONFIG_FIELDS = {
+    "mode",
+    "input_data",
+    "reference_path",
+    "ep",
+    "precision",
+    "quant",
+    "optimize",
+    "analyze",
+    "max_optim_iterations",
+    "shape_config",
+    "export_overrides",
+    "allow_unsupported_nodes",
+    "skip_build",
+    "use_cache",
+    "rebuild",
+}
+
+
+def _validate_pytorch_runtime_options(
+    ctx: click.Context,
+    cfg: WinMLEvaluationConfig,
+    config_fields: set[str],
+) -> None:
+    """Reject options whose semantics require ONNX export or ONNX Runtime."""
+    if cfg.device.lower() not in ("auto", "cpu", "gpu"):
+        raise click.UsageError(
+            f"--device {cfg.device} is not supported with --runtime pytorch; use auto, cpu, or gpu."
+        )
+    incompatible = [
+        flag
+        for param_name, flag in _PYTORCH_RUNTIME_INCOMPATIBLE_OPTIONS.items()
+        if cli_utils.is_cli_provided(ctx, param_name)
+    ]
+    incompatible.extend(
+        f"eval.{field}"
+        for field in sorted(config_fields & _PYTORCH_RUNTIME_INCOMPATIBLE_CONFIG_FIELDS)
+    )
+    if incompatible:
+        raise click.UsageError(
+            "--runtime pytorch cannot be combined with incompatible options: "
+            f"{', '.join(incompatible)}."
+        )
+
+
 def _resolve_model(
     cfg: WinMLEvaluationConfig,
     model: tuple[str, ...],
@@ -520,6 +631,8 @@ def _resolve_model(
     allow_missing_model_id: bool = False,
 ) -> None:
     """Resolve ``-m`` / ``--model-id`` into ``cfg.model_path`` / ``cfg.model_id``."""
+    if not model and model_id is None and (cfg.model_path is not None or cfg.model_id is not None):
+        return
     model_path, resolved_id = _resolve_model_path(
         model=model, model_id=model_id, allow_missing_model_id=allow_missing_model_id
     )
@@ -620,6 +733,17 @@ def _apply_export_overrides(
 
 def _resolve_device(cfg: WinMLEvaluationConfig) -> None:
     """Resolve ``'auto'`` → concrete device string on *cfg* in place."""
+    if cfg.runtime == "pytorch":
+        from ..loader import resolve_native_device
+
+        try:
+            resolved = resolve_native_device(cfg.device)
+        except ValueError as error:
+            raise click.UsageError(str(error)) from error
+        cfg._auto_device_selected = cfg.device.lower() == "auto"
+        cfg.device = resolved.name
+        return
+
     if cfg.device and cfg.device.lower() != "auto":
         return
 
@@ -888,6 +1012,7 @@ def display_eval_report(result: EvalResult, console: Console) -> None:
     # Info section
     console.print()
     console.print(f"[dim]Task:[/dim]       {cfg.task}")
+    console.print(f"[dim]Runtime:[/dim]    {cfg.runtime}")
     console.print(f"[dim]Device:[/dim]     {cfg.device}")
     if cfg.input_data:
         console.print(f"[dim]Input data:[/dim] {cfg.input_data}")

src/winml/modelkit/eval/__init__.py

@@ -14,7 +14,7 @@
 from typing import TYPE_CHECKING, Any
 
 from .base_evaluator import WinMLEvaluator
-from .config import DatasetConfig, WinMLEvaluationConfig
+from .config import DatasetConfig, EvalRuntime, WinMLEvaluationConfig
 from .evaluate import EvalResult, evaluate, get_evaluator_class
 
 
@@ -47,57 +47,50 @@
 
 _LAZY_ATTRS: dict[str, str] = {
     # Evaluators
-    "WinMLDepthEstimationEvaluator":
-        ".depth_estimation_evaluator:WinMLDepthEstimationEvaluator",
-    "WinMLFeatureExtractionEvaluator":
-        ".feature_extraction_evaluator:WinMLFeatureExtractionEvaluator",
-    "WinMLFillMaskEvaluator":
-        ".fill_mask_evaluator:WinMLFillMaskEvaluator",
-    "WinMLImageFeatureExtractionEvaluator":
-        ".image_feature_extraction_evaluator:WinMLImageFeatureExtractionEvaluator",
-    "WinMLImageSegmentationEvaluator":
-        ".image_segmentation_evaluator:WinMLImageSegmentationEvaluator",
-    "WinMLImageToTextEvaluator":
-        ".image_to_text_evaluator:WinMLImageToTextEvaluator",
-    "WinMLKeypointDetectionEvaluator":
-        ".keypoint_detection_evaluator:WinMLKeypointDetectionEvaluator",
-    "WinMLObjectDetectionEvaluator":
-        ".object_detection_evaluator:WinMLObjectDetectionEvaluator",
-    "WinMLQuestionAnsweringEvaluator":
-        ".question_answering_evaluator:WinMLQuestionAnsweringEvaluator",
-    "WinMLTextClassificationEvaluator":
-        ".text_classification_evaluator:WinMLTextClassificationEvaluator",
-    "WinMLTextGenerationEvaluator":
-        ".text_generation_evaluator:WinMLTextGenerationEvaluator",
-    "WinMLTokenClassificationEvaluator":
-        ".token_classification_evaluator:WinMLTokenClassificationEvaluator",
-    "WinMLZeroShotClassificationEvaluator":
-        ".zero_shot_classification_evaluator:WinMLZeroShotClassificationEvaluator",
-    "WinMLZeroShotImageClassificationEvaluator":
-        ".zero_shot_image_classification_evaluator:WinMLZeroShotImageClassificationEvaluator",
-    "TensorSimilarityEvaluator":
-        ".tensor_similarity_evaluator:TensorSimilarityEvaluator",
+    "WinMLDepthEstimationEvaluator": ".depth_estimation_evaluator:WinMLDepthEstimationEvaluator",
+    "WinMLFeatureExtractionEvaluator": (
+        ".feature_extraction_evaluator:WinMLFeatureExtractionEvaluator"
+    ),
+    "WinMLFillMaskEvaluator": ".fill_mask_evaluator:WinMLFillMaskEvaluator",
+    "WinMLImageFeatureExtractionEvaluator": (
+        ".image_feature_extraction_evaluator:WinMLImageFeatureExtractionEvaluator"
+    ),
+    "WinMLImageSegmentationEvaluator": (
+        ".image_segmentation_evaluator:WinMLImageSegmentationEvaluator"
+    ),
+    "WinMLImageToTextEvaluator": ".image_to_text_evaluator:WinMLImageToTextEvaluator",
+    "WinMLKeypointDetectionEvaluator": (
+        ".keypoint_detection_evaluator:WinMLKeypointDetectionEvaluator"
+    ),
+    "WinMLObjectDetectionEvaluator": ".object_detection_evaluator:WinMLObjectDetectionEvaluator",
+    "WinMLQuestionAnsweringEvaluator": (
+        ".question_answering_evaluator:WinMLQuestionAnsweringEvaluator"
+    ),
+    "WinMLTextClassificationEvaluator": (
+        ".text_classification_evaluator:WinMLTextClassificationEvaluator"
+    ),
+    "WinMLTextGenerationEvaluator": ".text_generation_evaluator:WinMLTextGenerationEvaluator",
+    "WinMLTokenClassificationEvaluator": (
+        ".token_classification_evaluator:WinMLTokenClassificationEvaluator"
+    ),
+    "WinMLZeroShotClassificationEvaluator": (
+        ".zero_shot_classification_evaluator:WinMLZeroShotClassificationEvaluator"
+    ),
+    "WinMLZeroShotImageClassificationEvaluator": (
+        ".zero_shot_image_classification_evaluator:WinMLZeroShotImageClassificationEvaluator"
+    ),
+    "TensorSimilarityEvaluator": ".tensor_similarity_evaluator:TensorSimilarityEvaluator",
     # Metrics (defer numpy / scipy / torch / torchmetrics until first use)
-    "ClassificationMetric":
-        ".metrics.classification:ClassificationMetric",
-    "DepthMetric":
-        ".metrics.depth:DepthMetric",
-    "KeypointAPMetric":
-        ".metrics.keypoint:KeypointAPMetric",
-    "IGNORE_INDEX":
-        ".metrics.mean_iou:IGNORE_INDEX",
-    "KNNAccuracyMetric":
-        ".metrics.knn_accuracy:KNNAccuracyMetric",
-    "MAPMetric":
-        ".metrics.mean_average_precision:MAPMetric",
-    "MeanIoUMetric":
-        ".metrics.mean_iou:MeanIoUMetric",
-    "PseudoPerplexityMetric":
-        ".metrics.pseudo_perplexity:PseudoPerplexityMetric",
-    "SpearmanCorrelationMetric":
-        ".metrics.spearman_correlation:SpearmanCorrelationMetric",
-    "TopKAccuracyMetric":
-        ".metrics.top_k_accuracy:TopKAccuracyMetric",
+    "ClassificationMetric": ".metrics.classification:ClassificationMetric",
+    "DepthMetric": ".metrics.depth:DepthMetric",
+    "KeypointAPMetric": ".metrics.keypoint:KeypointAPMetric",
+    "IGNORE_INDEX": ".metrics.mean_iou:IGNORE_INDEX",
+    "KNNAccuracyMetric": ".metrics.knn_accuracy:KNNAccuracyMetric",
+    "MAPMetric": ".metrics.mean_average_precision:MAPMetric",
+    "MeanIoUMetric": ".metrics.mean_iou:MeanIoUMetric",
+    "PseudoPerplexityMetric": ".metrics.pseudo_perplexity:PseudoPerplexityMetric",
+    "SpearmanCorrelationMetric": ".metrics.spearman_correlation:SpearmanCorrelationMetric",
+    "TopKAccuracyMetric": ".metrics.top_k_accuracy:TopKAccuracyMetric",
 }
 
 
@@ -124,6 +117,7 @@ def __dir__() -> list[str]:
     "DatasetConfig",
     "DepthMetric",
     "EvalResult",
+    "EvalRuntime",
     "KNNAccuracyMetric",
     "KeypointAPMetric",
     "MAPMetric",