judging: reject unrenderable prompt templates at construction instead of aborting the run - #39
Open
ChrisW09 wants to merge 1 commit into
Open
judging: reject unrenderable prompt templates at construction instead of aborting the run#39ChrisW09 wants to merge 1 commit into
ChrisW09 wants to merge 1 commit into
Conversation
A custom prompt_template containing literal braces (a JSON example —
one of the most common judge-template patterns) or a typo'd placeholder
passed check_template_placeholders and then raised a bare KeyError at
the first score() call. The KeyError escaped LLMJudge.score (rendering
happens before the try, which only catches LLMError anyway), propagated
through the bare asyncio.gather in judge_results, and aborted the
entire judging phase — after the possibly expensive answer phase had
already been paid for.
check_template_placeholders now trial-renders the template with dummy
values and raises a ValueError explaining {{ }} escaping; both
documented template paths (Rubric and LLMJudge) validate at
construction. build_judge_prompt wraps its real format call the same
way as a guard for templates that bypass validation.
Fixes #15
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #15.
A custom
prompt_templatewith literal braces passescheck_template_placeholders(all required placeholders present) but explodes at score time:The
KeyErrorescapesLLMJudge.score(rendering happens before thetry, which only catchesLLMError), propagates through the bareasyncio.gatherinjudge_results, and aborts the whole judging phase — after the answer phase already spent its tokens. Same for any typo'd placeholder ({questions}).Fix:
check_template_placeholdersnow trial-renders the template with dummy values for every supported placeholder and raises aValueErrorexplaining{{ }}escaping and listing the supported placeholders. Both documented template paths (Rubric.prompt_template,LLMJudge.prompt_template) call this at construction, so the failure surfaces at study-definition time.build_judge_promptwraps its real.formatthe same way, so a template that bypassed validation produces one clear actionable error instead of a bareKeyError.Regression tests: JSON-example template rejected on both paths with an "escaped" hint, typo'd placeholder rejected, and a correctly
{{ }}-escaped JSON template renders as expected.Test: full cafe-core suite — 110 passed (1 pre-existing failure = #4, fix in PR #34).
🤖 Generated with Claude Code