Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
a42b5aa
docs: add Python core redesign prompt
jon-devlapaz Aug 15, 2026
9a18478
feat: replace evaluator core with python skill
jon-devlapaz Aug 15, 2026
d685d9b
feat: lock structured rubric tasks
jon-devlapaz Aug 15, 2026
96fd1d0
feat: add provisional Codex rubric judge
jon-devlapaz Aug 15, 2026
116bd3c
docs: describe provisional rubric judging
jon-devlapaz Aug 15, 2026
50a9d8c
docs: record provisional judge decision
jon-devlapaz Aug 15, 2026
c296e90
docs: lock run-local Codex home isolation
jon-devlapaz Aug 15, 2026
6a0324c
feat: isolate live Codex home from the host
jon-devlapaz Aug 15, 2026
8782fa0
docs: describe run-local Codex home
jon-devlapaz Aug 15, 2026
9f5c3ac
feat: add blinded pairwise rubric comparison
jon-devlapaz Aug 15, 2026
c0d981a
docs: describe blinded pairwise judging
jon-devlapaz Aug 15, 2026
5470e57
feat: make reports and exit status quality-aware
jon-devlapaz Aug 15, 2026
a121abe
docs: describe quality-aware reports and exit codes
jon-devlapaz Aug 15, 2026
494aa0f
feat: add human-labeled pairwise calibration
jon-devlapaz Aug 15, 2026
93da333
docs: describe pairwise judge calibration
jon-devlapaz Aug 15, 2026
29a89a2
fix: treat missing Codex model identity as unattested
jon-devlapaz Aug 15, 2026
f448459
docs: record live calibrate and unattested model identity
jon-devlapaz Aug 15, 2026
b83b928
docs: hand off the Karpathy hill-climb as Phase 2
jon-devlapaz Aug 15, 2026
c0a818a
feat: bind accepted calibration into rubric runs
jon-devlapaz Aug 16, 2026
8026f76
docs: document calibration-bound rubric runs
jon-devlapaz Aug 16, 2026
a3c0d72
ci: fix setup-python action pin
jon-devlapaz Aug 16, 2026
cae2636
fix: discard copied auth on initialization failure
jon-devlapaz Aug 16, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
56 changes: 6 additions & 50 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -13,41 +13,15 @@ jobs:
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7

- uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
- uses: actions/setup-python@e797f83bcb11b83ae66e0230d6156d7c80228e7c # v6.0.0
with:
go-version: "1.26.2"
python-version: "3.11"

- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7
with:
node-version: "22.20.0"
cache: npm

- name: Install pinned Node validation tools
run: npm ci --ignore-scripts


- name: Test Go evaluator
run: |
go test -race ./...
go vet ./...
test -z "$(gofmt -l cmd internal)"
- name: Test Python evaluator
run: python3 -m unittest discover -s tests -v

- name: Run evaluator healthcheck
run: bash skills/skill-eval-loop/scripts/healthcheck.sh

- name: Verify packaged platform binaries
run: |
test -x skills/skill-eval-loop/bin/linux-amd64/skill-eval-loop
test -x skills/skill-eval-loop/bin/linux-arm64/skill-eval-loop
test -x skills/skill-eval-loop/bin/darwin-amd64/skill-eval-loop
test -x skills/skill-eval-loop/bin/darwin-arm64/skill-eval-loop
skills/skill-eval-loop/bin/linux-amd64/skill-eval-loop healthcheck \
--skill-dir skills/skill-eval-loop
rebuilt="$RUNNER_TEMP/skill-eval-loop-linux-amd64"
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build \
-buildvcs=false -trimpath -ldflags='-s -w' \
-o "$rebuilt" ./cmd/skill-eval-loop
cmp "$rebuilt" skills/skill-eval-loop/bin/linux-amd64/skill-eval-loop
run: skills/skill-eval-loop/scripts/healthcheck.sh

- name: Check whitespace
run: git diff --check "$(git hash-object -t tree /dev/null)" HEAD
Expand All @@ -60,15 +34,6 @@ jobs:
exit 1
fi

- name: List installable skills
run: |
./node_modules/.bin/skills add . --list | tee /tmp/skills-list.txt
grep -q "skill-eval-loop" /tmp/skills-list.txt
test "$(find skills -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ')" = "1"

- name: Check README links
run: ./node_modules/.bin/markdown-link-check README.md

standalone-package:
name: Verify standalone skill (${{ matrix.platform }})
strategy:
Expand All @@ -90,7 +55,7 @@ jobs:
- name: Copy only the installed skill payload
run: cp -R skills/skill-eval-loop "$RUNNER_TEMP/skill-eval-loop"

- name: Run the public launcher without development tools
- name: Run the public launcher with only system Python
working-directory: ${{ runner.temp }}
run: |
env -i \
Expand Down Expand Up @@ -139,17 +104,8 @@ jobs:
"$tink" init --no-zen --no-tink-skills --no-manage-tink
"$tink" skill add jon-devlapaz/skill-eval-loop --skill skill-eval-loop
"$tink" skill check

"$tink" skill lock
"$tink" skill verify

evaluator="$project_root/.agents/skills/skill-eval-loop/scripts/skill-eval-loop"
"$evaluator" healthcheck
"$evaluator" audit \
--skill-path "$GITHUB_WORKSPACE/conformance/scenarios/fixtures/recommend-explicit/skill"

chmod a+x .agents/skills/skill-eval-loop/SKILL.md
if "$tink" skill verify; then
echo "Tink accepted a payload incompatible with its lock" >&2
exit 1
fi
4 changes: 4 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,3 +3,7 @@
## Maintainability

Follow the maintainability principles in [ZEN.md](ZEN.md).

## Next change

Continue **Phase 2** in [tasks/plan.md](tasks/plan.md) (checklist: [tasks/todo.md](tasks/todo.md)). That is a CI-gated hill climb on the existing evaluator. Do not start by splitting `skill_eval_loop.py`.
Loading