fix: TODO batch — grader honesty, cost reporting, curated changelog

[tardigrde]

Summary

Works through the actionable items in TODO.md (skipping the subagent-evals generalization, which is deliberately deferred):

Grader honesty (fix)

• LLMGrader.grade: grading errors and missing per-assertion results are now skipped with a stderr warning instead of counted as failures, so grader problems never masquerade as agent failures.
• Workspace resolution for file-based checks is shared between the deterministic and LLM graders and warns (once per run dir) when it falls back to guessing from the output-dir layout (e.g. re-grading after the workspace was deleted).
• fix-failing-tests negative-control: the grader-run-dependent "output contains the source code" assertion is reworded to the behavior that matters (file read and displayed/described; not modified and no tests run are separate assertions).

Cost + validate (feat)

• benchmark.json gains per-config cost_usd mean/stddev and per-agent cost deltas; report shows a Cost (USD) column in both table and markdown formats. Old benchmark/timing files without the field still load.
• validate now classifies every assertion (classify_assertion, the same routing the deterministic grader uses) and lists the ones that will fall through to the LLM rubric, instead of that happening silently at run time.

Changelog / release hygiene

• Curated pre-0.4 CHANGELOG entries restored (from v0.4.0~1), converted to PSR-style ## vX.Y.Z (date) headings so PSR's update-mode dedup check recognizes them.
• python-semantic-release now runs in insertion-only mode = "update" with an insertion flag; default PSR templates render summary lines only (no commit bodies / Co-Authored-By trailers). Verified end-to-end in a throwaway clone (new release inserted at the flag, summary lines only, curated history untouched, no duplicate sections).
• Version single-sourced from pyproject.toml (PSR version_variables removed; __init__.py falls back to 0.0.0 when not installed).
• TODO.md updated: items done in this batch ticked, plus several items verified as already done on main (--show-evidence, --version, init scaffold, assertion-pattern docs).

Testing

• pytest: 232 passed (includes the free-tier e2e pipeline test)
• ruff check + ruff format --check: clean
• PSR changelog rendering verified manually in a scratch clone with a trailer-carrying feat: commit

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces several improvements to the evaluation framework, including adding a 'Cost (USD)' column to benchmark reports, documenting and validating assertion patterns, and gracefully skipping (rather than failing) assertions when the LLM grader fails or drops them. It also refactors versioning to single-source from pyproject.toml and configures python-semantic-release with custom templates to prevent git trailers from leaking into the changelog. Feedback on the changes suggests normalizing assertion texts (e.g., lowercasing and stripping trailing periods) before matching them against the LLM grader's response to make the matching process more robust against minor formatting variations.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

The LLM grader matching is currently doing an exact string match (a not in returned_texts). LLMs frequently make minor formatting changes to the assertion texts in their JSON responses, such as changing casing, adding/removing trailing periods, or normalizing whitespace. This can cause assertions to be falsely classified as missing and skipped.

We should normalize both the returned texts and the assertions (e.g., lowercasing, stripping whitespace, and removing trailing periods) before performing the membership check to make the matching much more robust.

Suggested change

	returned_texts.add(r["text"])
	for a in assertions:
	if a not in returned_texts:
	results.append(
	AssertionResult(
	text=a,
	passed=False,
	evidence="LLM grader did not return a result for this assertion",
	method="llm",
	)
	missing = [a for a in assertions if a not in returned_texts]
	returned_texts.add(r["text"].lower().strip().rstrip('.'))
	missing = [a for a in assertions if a.lower().strip().rstrip('.') not in returned_texts]

[tardigrde]

Done in 1e52852: normalization helper (lowercase, collapse whitespace, strip trailing period) applied to both sides of the membership check.