Isolate skill evaluations and prepare v1 release#6
Conversation
There was a problem hiding this comment.
Code Review
This pull request updates the evaluation harness to protocol version 3, introducing isolated execution workspaces to prevent skill contamination and requiring adapters to return an isolation attestation. It adds a --work-root option, resolves runner command file arguments to absolute paths, and validates that no same-name skills exist in ancestor paths. The feedback highlights an issue in prepare_work_root where a newly created, user-specified work root is not cleaned up if subsequent validation checks fail, and suggests tracking directory creation to ensure proper cleanup.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
Summary
Why
A report path under the repository could make no-skill baselines discover the repository's
.agentsor.claudemirrors. Separating execution from reporting and requiring adapter attestation prevents those contaminated runs from being accepted.Verification
python scripts/verify_all.pyon Windows: 81 tests, 1 POSIX-only skippython3 scripts/verify_all.pyin WSL with Python 3.14.4: 81 testspython -m ruff check .reports/and auto-cleaned work root outside the repositorygh skill publish --dry-run .