Skip to content

Isolate skill evaluations and prepare v1 release#6

Merged
dd3ok merged 3 commits into
mainfrom
codex/eval-isolation-release
Jul 10, 2026
Merged

Isolate skill evaluations and prepare v1 release#6
dd3ok merged 3 commits into
mainfrom
codex/eval-isolation-release

Conversation

@dd3ok

@dd3ok dd3ok commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Summary

  • execute evaluation workspaces outside the repository and reject known same-name skill discovery ancestors
  • require protocol v3 adapter isolation attestations and archive only bounded report artifacts
  • document pinned v1.0.0 installation and the isolated evaluation contract in English and Korean

Why

A report path under the repository could make no-skill baselines discover the repository's .agents or .claude mirrors. Separating execution from reporting and requiring adapter attestation prevents those contaminated runs from being accepted.

Verification

  • python scripts/verify_all.py on Windows: 81 tests, 1 POSIX-only skip
  • python3 scripts/verify_all.py in WSL with Python 3.14.4: 81 tests
  • python -m ruff check .
  • 36-run fixture smoke with report output under reports/ and auto-cleaned work root outside the repository
  • gh skill publish --dry-run .

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the evaluation harness to protocol version 3, introducing isolated execution workspaces to prevent skill contamination and requiring adapters to return an isolation attestation. It adds a --work-root option, resolves runner command file arguments to absolute paths, and validates that no same-name skills exist in ancestor paths. The feedback highlights an issue in prepare_work_root where a newly created, user-specified work root is not cleaned up if subsequent validation checks fail, and suggests tracking directory creation to ensure proper cleanup.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment thread scripts/run_skill_eval.py
@dd3ok dd3ok marked this pull request as ready for review July 10, 2026 09:08
@dd3ok dd3ok enabled auto-merge (squash) July 10, 2026 09:08
@dd3ok dd3ok merged commit 124a903 into main Jul 10, 2026
4 checks passed
@dd3ok dd3ok deleted the codex/eval-isolation-release branch July 10, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant