Skip to content

Filter compare views by tag #1041

Description

@christso

Follow-up to #1037 / PR #1040, which introduced multi-valued tags as retroactive annotation on individual runs.

Objective

Let users narrow both the Aggregated compare matrix and the Per-run compare table to only runs matching a selected set of tags, so they can answer questions like "how did runs tagged v2-prompt fare across all targets?" without manually de-selecting rows.

Design latitude

Tags are multi-valued, so they cannot cleanly replace experiment as a matrix dimension (doing so would double-count runs and break row totals). The recommended direction is filter, not dimension: tag chips above the compare view narrow which runs contribute to cells[] and runs[], while grouping stays (experiment, target) for the aggregated matrix.

This matches the Langfuse, W&B, and GitHub Issues convention where tags/labels are a filter facet rather than a pivot axis.

Open questions for the implementer:

  • AND vs OR semantics. OR ("any of the selected tags matches") is the industry default and easiest to explain; AND is also defensible for power users. Pick one for v1, note the choice in the UI.
  • Which tags to surface. The filter chip row should probably list every distinct tag currently attached to any run in the active workspace (from compareResponse.runs[].tags). Alphabetical, with a count suffix (baseline · 3) if useful.
  • Composition with per-run view. The filter should apply to both modes so a user can flip between Aggregated and Per-run without losing their filter state.
  • Empty state. If the filter yields zero cells, show a notice like "No runs match baseline + v2-prompt" with a "Clear filter" button.
  • URL/query-state persistence. Optional nicety: reflect the filter in the ?tags=... query string so reloads and shared links preserve the view.

Acceptance signals

  • A user with ≥2 runs and ≥2 distinct tags can click a tag chip above the compare view and see both the matrix and the per-run table narrow to only matching runs.
  • Clearing the filter restores the full view.
  • /api/compare?tags=baseline,v2-prompt on the backend returns cells[] and runs[] filtered to runs carrying at least one matching tag (OR semantics).
  • No regression in the unfiltered /api/compare call.
  • Typecheck, lint, unit tests, and studio build pass.
  • Manual red/green UAT documented on the PR (screenshots of filtered vs unfiltered compare views).

Non-goals

  • Not replacing experiment with tag as a matrix column (fundamentally unclean for multi-valued attributes).
  • Not a full tag management UI (bulk rename, merge, delete-by-tag across runs). Point changes go through the existing per-run chip editor.
  • Not tag-as-eval-metadata in the YAML schema — tags are mutable sidecars, not part of the run manifest.
  • Not colored tags, tag groups, tag descriptions, or any tag-about-tags metadata. Follow YAGNI until users ask.

Implementation sketch

Backend (apps/cli/src/commands/results/serve.ts):

// In handleCompare, after resolving the tags sidecar for each run:
const filterTags = (c.req.query('tags') ?? '').split(',').filter(Boolean);
if (filterTags.length > 0) {
  const runTags = tagsEntry?.tags ?? [];
  if (!filterTags.some((t) => runTags.includes(t))) continue;
}

One early-return, no schema change. Same treatment for the benchmark-scoped endpoint.

Frontend (apps/studio/src/components/CompareTab.tsx):

  • New state: const [filterTags, setFilterTags] = useState<string[]>([])
  • Chip row above the content switch listing distinct tags from data.runs, active ones rendered with the cyan-filled variant (reuse the chip classNames from the existing TagsEditor so styling stays consistent)
  • Pass filterTags into the React Query key: ['compare', { tags: filterTags }] so refetch is automatic when the filter changes
  • Empty-state notice when filter yields zero cells
  • Keep the filter state local to CompareTab for v1; promote to URL query string only if you tackle the shareable-link nicety

Studio design reference: before writing new UI, skim apps/studio/DESIGN.md (canonical Tailwind patterns + the do/don't list — in particular the single-cyan-accent rule and the mandatory PassRatePill reuse).

Related

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

in-progressClaimed by an agent — do not duplicate workwuiRelates to the browser dashboard / web UI runtime

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions