Skip to content

feat: wire --rubrics flag to RubricConfig.load_bundle() - #178

Draft
luca-belli wants to merge 1 commit into
feat/rubric-bundle-manifest-helperfrom
feat/wire-rubrics-cli-flag
Draft

feat: wire --rubrics flag to RubricConfig.load_bundle()#178
luca-belli wants to merge 1 commit into
feat/rubric-bundle-manifest-helperfrom
feat/wire-rubrics-cli-flag

Conversation

@luca-belli

Copy link
Copy Markdown
Collaborator

Summary

  • Stacked on feat: add RubricConfig.load_bundle() for rubric bundle manifests #177 (adds RubricConfig.load_bundle()); base branch is that PR's branch, not main
  • judge.py's --rubrics/-r flag was declared but never read; main() now loads args.rubrics[0] via RubricConfig.load_bundle() instead of hardcoding RubricConfig.load(rubric_folder="data")
  • Default changed from data/rubric.tsv to data/rubric_manifest.json (the production manifest added in feat: add RubricConfig.load_bundle() for rubric bundle manifests #177)
  • Passing more than one --rubrics value now prints a warning to stderr and uses only the first (multi-rubric support isn't implemented yet; kept the flag's list shape forward-compatible)
  • run_pipeline.py has its own --rubrics flag that it forwards straight into judge.py's main() -- updated its default the same way so the full pipeline doesn't break on its own default
  • Added tests/fixtures/rubric_manifest_multi_row.json, a second test bundle with genuinely different rubric content, used by a new end-to-end test proving --rubrics actually drives which bundle loads (not a hardcoded no-op)

Test plan

  • uv run pytest -m "not live" passes (965 passed)
  • New test: --rubrics <manifest> loads distinct rubric content for two different manifests (real RubricConfig.load_bundle(), no mocking of that call)
  • New test: multiple --rubrics values warns on stderr and uses only the first
  • Updated existing unit/integration tests that asserted the old data/rubric.tsv default or mocked RubricConfig.load
  • uv run ruff check / uv run ruff format --check clean

@luca-belli
luca-belli marked this pull request as draft July 23, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant