feat: redesign evaluator core in Python - #1
Merged
Conversation
Judge each condition only after deterministic gates pass, fail closed to unknown, and label same-provider results as non-independent. Co-authored-by: Cursor <cursoragent@cursor.com>
Document the live judge path, fail-closed unknown status, and same-provider limit so the public contract matches the runner. Co-authored-by: Cursor <cursoragent@cursor.com>
Mark Task 3 complete under the approved OpenAI-only judge pairing and leave pairwise comparison unclaimed. Co-authored-by: Cursor <cursoragent@cursor.com>
Require a private $output/codex-home with auth.json copy only, so live runs do not inherit the host Codex home. Co-authored-by: Cursor <cursoragent@cursor.com>
Create $output/codex-home for the run, copy only auth.json while Codex executes, and drop the host-home skill gate. Co-authored-by: Cursor <cursoragent@cursor.com>
Document the private output Codex home and mark Task 3b complete under fake-adapter evidence. Co-authored-by: Cursor <cursoragent@cursor.com>
Judge anonymized A/B candidates after per-output scores, keep the mapping out of the prompt, and restore labels only in retained evidence. Co-authored-by: Cursor <cursoragent@cursor.com>
Document anonymized A/B comparison and mark Task 4 complete under fake-adapter evidence. Co-authored-by: Cursor <cursoragent@cursor.com>
Separate runner validity from quality completeness, expose every dimension, and use exit 0/1/2 so an aggregate winner cannot hide unknown or inconsistent evidence. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Lock known-better, known-worse, and tie cases with rationale, score a blinded judge against that threshold, and retain disagreements. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Live exec traces from Codex 0.147.0 never emit system.init.model. Keep mismatch fail-closed, but parse judgments when identity is absent. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Leave the next agent a locked objective and ordered tasks on the current evaluator instead of a clean-tree guess. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR covers the full original 20-commit
python-core-redesignstack fromorigin/main, not only Task 8, plus one CI-only commit correcting the immutableactions/setup-pythonv6.0.0 pin.Validation
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v— 29 tests passedpy_compilewith bytecode redirected outside the repository — passedgit diff --check HEAD~2 HEAD— passed for the final Task 8 commitsskills/skill-eval-loop/scripts/healthcheck.sh—valid: trueorigin/main— 0 behind / 20 ahead before the CI-only follow-up; dry merge completed without conflict diagnosticsEvidence boundaries
provisional_non_independent; this PR does not claim independent evaluation.calibration.jsonplus its original absolute fixture path as the trust root. The runner verifies internal consistency and hashes but does not authenticate the origin of raw prompt, response, trace, or stderr artifacts.