Skip to content

Feat/text to image eval primitives: add prompt-driven evaluation primitives (PromptDataset + CLIPScore + Recall@K + MRR) #1284

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
AChenreddy24 wants to merge 8 commits into
base: main
Choose a base branch
from
Open

Feat/text to image eval primitives: add prompt-driven evaluation primitives (PromptDataset + CLIPScore + Recall@K + MRR) #1284

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
d726e14
feat(datasets): add PromptDataset primitive for prompt-driven evaluators
AChenreddy24 Jul 20, 2026
d0d9c6a
feat(eval): add CLIPScoreMetric for text-image alignment
AChenreddy24 Aug 5, 2026
b97b07b
feat(eval): add RecallAtKMetric for retrieval and ranked-neighbor eva…
AChenreddy24 Aug 5, 2026
6fba49f
feat(eval): add MeanReciprocalRankMetric for retrieval evaluation
AChenreddy24 Aug 5, 2026
253a2a7
feat(eval): report Recall@K and MRR from image-feature-extraction eva…
AChenreddy24 Aug 5, 2026
f191b86
feat(datasets): add filter/sample to PromptDataset with _derive exten…
AChenreddy24 Aug 5, 2026
db4329a
Merge branch 'main' into feat/text-to-image-eval-primitives
AChenreddy24 Aug 5, 2026
daf8a47
fix(datasets): add return type to PromptDataset.__iter__ for mypy
AChenreddy24 Aug 5, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions src/winml/modelkit/datasets/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
from .mask_generation import MaskGenerationDataset
from .object_detection import DEFAULT_OBJECT_DETECTION_SIZE, ObjectDetectionDataset
from .processor_utils import get_image_processor_config
from .prompt_dataset import PromptDataset, PromptRecord

@DingmaomaoBJTU Qiong Wu (qiowu) (DingmaomaoBJTU) Aug 6, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since these are being exposed from the package root, please also add PromptDataset and PromptRecord to __all__ below. Otherwise from winml.modelkit.datasets import * and code/docs that rely on the declared public API won't include the new prompt dataset primitives, unlike the other dataset classes exported here.

@zhenchaoni Zhenchao Ni (zhenchaoni) Aug 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we wire these primitives into the evaluator flow in this PR? PromptDataset and CLIPScoreMetric are currently only exported; no evaluator or task registration selects them or runs inference through the model abstraction, so winml eval cannot use this new functionality. If this PR is intentionally primitives-only, that scope should be made explicit and the evaluator integration tracked separately.

from .random_dataset import RandomDataset
from .text import TextDataset

Expand Down
21 changes: 15 additions & 6 deletions src/winml/modelkit/datasets/base.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,10 @@ class BaseTaskDataset(ABC):
properties are readonly to ensure dataset immutability and thread safety.

Attributes:
model_name: HuggingFace model identifier or local model path
model_name: HuggingFace model identifier or local model path.
Optional — task-agnostic data sources (e.g. prompt corpora used
by generative-model evaluators) may leave this ``None``. Most
calibration and task-oriented subclasses require it.
dataset_name: Dataset identifier (HF dataset or local path)
data_split: Dataset split to use (e.g., 'train', 'validation', 'test')
"""
Expand All @@ -35,7 +38,7 @@ class BaseTaskDataset(ABC):

def __init__(
self,

@zhenchaoni Zhenchao Ni (zhenchaoni) Aug 7, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should PromptDataset inherit from BaseTaskDataset if doing so requires making model_name optional for every task dataset? This weakens the base contract and shifts validation to all existing subclasses. A model-agnostic dataset abstraction, or a separate prompt dataset base, may preserve the current task-dataset invariant more safely.

model_name: str,
model_name: str | None = None,
dataset_name: str | None = None,
max_samples: int | None = None,
data_split: str | None = None,
Expand All @@ -44,7 +47,10 @@ def __init__(
"""Initialize dataset with readonly properties.

Args:
model_name: HuggingFace model identifier or path
model_name: HuggingFace model identifier or path. Optional —
task-agnostic subclasses may pass ``None``. Task-oriented
subclasses that need a tokenizer/processor should validate
its presence themselves.
dataset_name: Dataset name (uses DEFAULT_DATASET if None)
max_samples: Maximum number of samples (None = use all)
data_split: Dataset split (None = let subclass decide)
Expand Down Expand Up @@ -78,8 +84,12 @@ def _initialize(self) -> None:

# Readonly properties
@property
def model_name(self) -> str:
"""Get the model name (readonly)."""
def model_name(self) -> str | None:
"""Get the model name (readonly).

``None`` for task-agnostic subclasses that are not bound to a
specific model.
"""
return self._model_name

@property
Expand All @@ -92,7 +102,6 @@ def data_split(self) -> str | None:
"""Get the dataset split (readonly)."""
return self._data_split


def __len__(self) -> int:
"""Return the number of samples in the dataset."""
if self._dataset is None:
Expand Down
Loading
Loading