Skip to content

judging: reject unrenderable prompt templates at construction instead of aborting the run - #39

Open
ChrisW09 wants to merge 1 commit into
mainfrom
fix/template-validation
Open

judging: reject unrenderable prompt templates at construction instead of aborting the run#39
ChrisW09 wants to merge 1 commit into
mainfrom
fix/template-validation

Conversation

@ChrisW09

Copy link
Copy Markdown
Collaborator

Fixes #15.

A custom prompt_template with literal braces passes check_template_placeholders (all required placeholders present) but explodes at score time:

prompt_template = 'Q: {question} A: {answer} S: {scale} Reply as JSON like {"grade": {grade}}'
# → KeyError: '"grade"' at the FIRST judge call

The KeyError escapes LLMJudge.score (rendering happens before the try, which only catches LLMError), propagates through the bare asyncio.gather in judge_results, and aborts the whole judging phase — after the answer phase already spent its tokens. Same for any typo'd placeholder ({questions}).

Fix:

  • check_template_placeholders now trial-renders the template with dummy values for every supported placeholder and raises a ValueError explaining {{ }} escaping and listing the supported placeholders. Both documented template paths (Rubric.prompt_template, LLMJudge.prompt_template) call this at construction, so the failure surfaces at study-definition time.
  • build_judge_prompt wraps its real .format the same way, so a template that bypassed validation produces one clear actionable error instead of a bare KeyError.
  • The missing-placeholder check stays a warning (those templates still render).

Regression tests: JSON-example template rejected on both paths with an "escaped" hint, typo'd placeholder rejected, and a correctly {{ }}-escaped JSON template renders as expected.

Test: full cafe-core suite — 110 passed (1 pre-existing failure = #4, fix in PR #34).

🤖 Generated with Claude Code

A custom prompt_template containing literal braces (a JSON example —
one of the most common judge-template patterns) or a typo'd placeholder
passed check_template_placeholders and then raised a bare KeyError at
the first score() call. The KeyError escaped LLMJudge.score (rendering
happens before the try, which only catches LLMError anyway), propagated
through the bare asyncio.gather in judge_results, and aborted the
entire judging phase — after the possibly expensive answer phase had
already been paid for.

check_template_placeholders now trial-renders the template with dummy
values and raises a ValueError explaining {{ }} escaping; both
documented template paths (Rubric and LLMJudge) validate at
construction. build_judge_prompt wraps its real format call the same
way as a guard for templates that bypass validation.

Fixes #15

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

judging: custom prompt template with literal braces passes validation but aborts the whole judging run

1 participant