Support unique parameters for multiple judges

[jgieringer]

Fix: -jep silently no-ops for unsupported providers in a mixed judge fleet

Context

judge.py -j supports a mixed fleet of judge models across providers (e.g. -j gpt-5.4:2 claude-sonnet-5:3), but -jep/--judge-model-extra-params parses into a single flat dict applied identically to every judge instance regardless of provider (judge.py:76-86, judge/runner.py:83-88).

Since v1.2.0 added provider-specific reasoning knobs (reasoning_effort for OpenAI, thinking_effort for Claude, thinking_level/thinking_budget for Gemini — see README.md:364-386), running -j gpt-5.4:2 claude-sonnet-5:3 -jep reasoning_effort=low silently drops reasoning_effort for the Claude instances. Both ChatAnthropic and ChatOpenAI set pydantic model_config = {"extra": "ignore", ...} (confirmed directly), so an unrecognized kwarg passed to ChatAnthropic(**llm_params) in ClaudeLLM.__init__ (llm_clients/claude_llm.py:136-205) is dropped with no error or warning. The gpt-5.4 judges get low reasoning effort; the claude-sonnet-5 judges quietly run without extended thinking, and nothing signals that happened. generate.py's -uep/-pep don't have this problem — each targets exactly one model, so there's no cross-provider ambiguity there.

Fix (per user decision): add per-model scoping to -jep so a single command can drive each provider's native reasoning param, and add a warning when a param is applied to a model whose provider doesn't recognize it — so the failure mode becomes visible instead of silent.

New CLI syntax

-jep/--judge-model-extra-params becomes nargs="+". Each entry is either:

• key=value[,key2=value2,...] — global, applies to every judge model (current behavior, unchanged when used alone)
• model:key=value[,key2=value2,...] — scoped, applies only to that judge model (must match a name passed to -j), merged on top of the global entries

uv run python judge.py -f output/{RUN} \
  -j gpt-5.4:2 claude-sonnet-5:3 \
  -jep "temperature=0" \
       "gpt-5.4:reasoning_effort=low" \
       "claude-sonnet-5:thinking_effort=low"

If a global (unscoped) key is a known reasoning knob (reasoning_effort, thinking_effort, thinking_level, thinking_budget) and doesn't match the provider of one of the -j models, print a warning naming the affected model and the correct param to use instead. Referencing a model: prefix that isn't one of the -j models raises a clear ValueError (typo protection).

Implementation

llm_clients/llm_factory.py — extract the existing model_lower if/elif provider-detection chain in create_llm (azure → ollama → claude → gpt/openai → gemini → endpoint) into a new LLMFactory.infer_provider(model_name: str) -> str staticmethod; create_llm calls it instead of duplicating the chain. Reused by the new warning check so provider detection has one source of truth.

judge/utils.py — add:

• parse_judge_model_extra_params_entry(entry: str) -> tuple[str | None, dict] — detects model: scoping (colon before the first =), returns (model_or_None, parsed_dict) via the existing parse_key_value_list.
• resolve_judge_model_extra_params(judge_models, global_params, per_model_overrides) -> Dict[str, Dict[str, Any]] — for each model in judge_models, merges global_params with that model's overrides; raises ValueError if per_model_overrides references a model not in judge_models; calls the mismatch-warning check; returns the per-model map.
• A small _REASONING_PARAM_PROVIDERS registry (reasoning_effort→openai, thinking_effort→claude, thinking_level/thinking_budget→gemini) plus a helper that prints a warning (print(..., file=sys.stderr), matching judge.py's existing advisory-message style) for any (model, param) pair whose provider doesn't match, via LLMFactory.infer_provider.
• Update build_judge_info(judge_models, judge_model_extra_params: Optional[Dict[str, Dict[str, Any]]]): when every model resolves to the same params (today's common case, including single-model runs), keep the exact current string format (no test or doc changes needed there). Only when models genuinely differ, embed each model's own param suffix after its own modelxcount segment, using single underscores (never __, which is reserved as a structural separator in build_evaluation_run_folder_path/parse_judge_info_from_run_folder).

judge.py — replace the plain type=parse_key_value_list on -jep with a custom argparse.Action that scans all provided entries, splits scoped vs. global via parse_judge_model_extra_params_entry, and sets two namespace attributes: judge_model_extra_params (global dict only — same shape/semantics as today, so existing parser-level tests need no changes) and judge_model_extra_params_by_model (scoped overrides, default {} via parser.set_defaults). In main(), right after judge_models = parse_judge_models(...), call resolve_judge_model_extra_params(judge_models, args.judge_model_extra_params, getattr(args, "judge_model_extra_params_by_model", {})) to get the per-model map, and use that (not args.judge_model_extra_params) for: the single-conversation LLMJudge(...) call (judge_model_extra_params=<map>[first_model]), the folder-mode judge_kwargs["judge_model_extra_params"] (pass the whole map), and the --resume mismatch check's build_judge_info(judge_models, <map>) call. Using getattr (not direct attribute access) keeps run_pipeline.py working unmodified — it builds its own argparse.Namespace for judge.py's main() (run_pipeline.py:570-583) and only sets judge_model_extra_params (flat), never the new attribute.

judge/runner.py — change the type of judge_model_extra_params on judge_conversations, batch_evaluate_with_individual_judges, and _create_evaluation_jobs from "one shared dict" to "Dict[model_name, Dict[str, Any]]". Only _create_evaluation_jobs needs a logic change: instead of putting the same object into every job tuple, use (judge_model_extra_params or {}).get(judge_model, {}) per job. The job-tuple shape (7 elements), _worker, _evaluate_single_conversation_with_judge, and _run_workers_with_queue are untouched — they already handle a single resolved dict per job.

Tests

• Update (mechanical, to match the new per-model contract):
  • tests/unit/judge/test_judge_cli.py:158,221 — main() now forwards the resolved per-model map, e.g. judge_model_extra_params={"gpt-4o": {}} instead of {}.
  • tests/unit/judge/test_runner_extra_params.py (test_batch_evaluate_accepts_extra_params, test_judge_conversations_accepts_extra_params, test_batch_evaluate_extra_params_with_multiple_conversations) — wrap the flat extra_params dict under the single model key used in each test's judge_models.
• Add:
  • CLI parsing tests for scoped entries (-jep "temperature=0" "gpt-5.4:reasoning_effort=low"), unknown-model-reference error, and that plain unscoped usage is unaffected.
  • resolve_judge_model_extra_params unit tests: merge precedence (override beats global), multiple models with different overrides, unknown-model ValueError.
  • Reasoning-knob mismatch warning test (e.g. capsys) for -j gpt-5.4 claude-sonnet-5 -jep reasoning_effort=low and confirm no warning when params are already correctly scoped.
  • build_judge_info tests: identical output to today when params are uniform/absent; new per-model-segment format when they differ, and confirm it doesn't introduce __.
• Verify unaffected: tests/integration/test_evaluation_runner.py:2019 (assert len(jobs[0]) == 7) — tuple shape isn't changing, but re-run to confirm.

Docs

• README.md:352-362 (multiple judge models) — mention -jep can be scoped per model.
• README.md:364-386 (Reasoning / extended thinking) — add the mixed-fleet example and note the mismatch warning.
• README.md:341-350 (extra-params folder-naming note) — clarify per-model encoding when params differ across models.
• Add an ## [Unreleased] bullet in CHANGELOG.md.

Verification

• uv run pytest -m "not live" (per /verify) — full suite green, including the updated/added tests above.
• Manual CLI smoke test against a small existing p_* run folder:

  uv run python judge.py -f <small_p_run> -j gpt-5.4 claude-sonnet-5 \
    -jep "temperature=0" "gpt-5.4:reasoning_effort=low"
  

  Confirm: no crash, a stderr warning fires for the unscoped case (-jep reasoning_effort=low with no scoping) but not for the scoped case, and --resume against the resulting folder still validates correctly.