Skip to content

feat: redesign evaluator core in Python - #1

Merged
jon-devlapaz merged 22 commits into
mainfrom
python-core-redesign
Aug 16, 2026
Merged

feat: redesign evaluator core in Python#1
jon-devlapaz merged 22 commits into
mainfrom
python-core-redesign

Conversation

@jon-devlapaz

@jon-devlapaz jon-devlapaz commented Aug 16, 2026

Copy link
Copy Markdown
Owner

Summary

  • replace the broad Go/multi-harness evaluator with a focused packaged Python 3 Codex evaluation loop
  • retain exact skill hashing, isolated control/treatment runs, run-local Codex homes, structured rubric grading, blinded pairwise comparison, quality-aware reports, and human-labeled calibration evidence
  • add Task 8 calibration binding: production calibration exercises both candidate orientations, rubric runs bind an accepted calibration fixture hash, missing calibration cannot complete quality evidence, and malformed or drifted bindings are runner-invalid
  • update the public skill instructions, minimum executable contract, Phase 2 plan, and checklist to match the implementation

This PR covers the full original 20-commit python-core-redesign stack from origin/main, not only Task 8, plus one CI-only commit correcting the immutable actions/setup-python v6.0.0 pin.

Validation

  • PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v — 29 tests passed
  • tracked Python py_compile with bytecode redirected outside the repository — passed
  • git diff --check HEAD~2 HEAD — passed for the final Task 8 commits
  • skills/skill-eval-loop/scripts/healthcheck.shvalid: true
  • pre-push ancestry check after fetching origin/main — 0 behind / 20 ahead before the CI-only follow-up; dry merge completed without conflict diagnostics
  • GitHub Actions on the final head — 12/12 checks passed across push and PR runs, including validate, four-platform standalone package verification, and Tink package verification

Evidence boundaries

  • OpenAI judging remains provisional_non_independent; this PR does not claim independent evaluation.
  • The recorded toy pilot was saturated and produced a pairwise tie. This PR makes no skill-quality winner claim.
  • Task 7 live verification on a named, locked, non-toy suite is deferred.
  • Tasks 9 and 10 are not started: there is not yet an honest CI quality gate or an independent judge/holdout.
  • Calibration binding uses the operator-controlled calibration.json plus its original absolute fixture path as the trust root. The runner verifies internal consistency and hashes but does not authenticate the origin of raw prompt, response, trace, or stderr artifacts.

jon-devlapaz and others added 22 commits August 14, 2026 20:00
Judge each condition only after deterministic gates pass, fail closed to unknown, and label same-provider results as non-independent.

Co-authored-by: Cursor <cursoragent@cursor.com>
Document the live judge path, fail-closed unknown status, and same-provider limit so the public contract matches the runner.

Co-authored-by: Cursor <cursoragent@cursor.com>
Mark Task 3 complete under the approved OpenAI-only judge pairing and leave pairwise comparison unclaimed.

Co-authored-by: Cursor <cursoragent@cursor.com>
Require a private $output/codex-home with auth.json copy only, so live runs do not inherit the host Codex home.

Co-authored-by: Cursor <cursoragent@cursor.com>
Create $output/codex-home for the run, copy only auth.json while Codex executes, and drop the host-home skill gate.

Co-authored-by: Cursor <cursoragent@cursor.com>
Document the private output Codex home and mark Task 3b complete under fake-adapter evidence.

Co-authored-by: Cursor <cursoragent@cursor.com>
Judge anonymized A/B candidates after per-output scores, keep the mapping out of the prompt, and restore labels only in retained evidence.

Co-authored-by: Cursor <cursoragent@cursor.com>
Document anonymized A/B comparison and mark Task 4 complete under fake-adapter evidence.

Co-authored-by: Cursor <cursoragent@cursor.com>
Separate runner validity from quality completeness, expose every
dimension, and use exit 0/1/2 so an aggregate winner cannot hide
unknown or inconsistent evidence.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Lock known-better, known-worse, and tie cases with rationale, score a
blinded judge against that threshold, and retain disagreements.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Live exec traces from Codex 0.147.0 never emit system.init.model. Keep
mismatch fail-closed, but parse judgments when identity is absent.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Leave the next agent a locked objective and ordered tasks on the current evaluator instead of a clean-tree guess.

Co-authored-by: Cursor <cursoragent@cursor.com>
@jon-devlapaz
jon-devlapaz merged commit f298be9 into main Aug 16, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant