Skip to content

Clarify evaluation loaders and unify composite model loading #1294

Description

@xieofxie

Context

_select_model_loader() in src/winml/modelkit/eval/evaluate.py selects several distinct loading contracts, but _ModelLoaderKind does not document what each kind means or who owns model/session construction.

Mask generation currently selects EVALUATOR_MANAGED: WinMLMaskGenerationEvaluator constructs its encoder and decoder ONNX Runtime sessions directly. This bypasses WinMLAutoModel because the composite model type is not registered. That makes the loading design task-specific and difficult to reuse for future evaluators backed by composite models.

Work

  • Document every _ModelLoaderKind value, including its input form, loading owner, returned model shape, build-pipeline behavior, and intended evaluation mode.
  • Investigate and design an architecture-agnostic composite-model loading path through WinMLCompositeModel / WinMLAutoModel.
  • Migrate mask-generation evaluation to the shared composite-model path if feasible.
  • Ensure future composite evaluators can reuse the same path without adding task- or architecture-specific branching to _select_model_loader().
  • Add pytest coverage for loader selection and composite-model loading behavior.

Acceptance criteria

  • Loader-kind semantics and precedence are clear at their declaration.
  • Mask-generation session ownership is either moved into a reusable composite model abstraction or the remaining blocker is explicitly documented.
  • The design does not hardcode model architecture, graph node, tensor, or layer names in shared loading code.
  • Existing ONNX, ONNX-to-HF comparison, ONNX-to-ONNX comparison, and GenAI loading behavior remains unchanged.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions