Skip to content

Remove unused skill evaluation harness#7

Merged
dd3ok merged 1 commit into
mainfrom
codex/remove-skill-eval-harness
Jul 10, 2026
Merged

Remove unused skill evaluation harness#7
dd3ok merged 1 commit into
mainfrom
codex/remove-skill-eval-harness

Conversation

@dd3ok

@dd3ok dd3ok commented Jul 10, 2026

Copy link
Copy Markdown
Owner

Summary

  • remove the unused cross-provider skill evaluation harness and its protocol-only fixtures/tests
  • keep lightweight trigger, forward-test, and fairness inputs outside the runtime skill
  • move repository test temporary directories out of reports/ so fresh checkouts do not require that directory
  • update English and Korean documentation to describe manual or vendor-provided evaluation

Why

The repository ships a backend development skill, while the harness had no real vendor adapter and could only prove its own protocol. Keeping it added about 2,400 lines of maintenance code without measuring skill quality or runtime token savings.

Impact

The installable skills/boring-backend/ package and its mirrors are unchanged. Runtime behavior and context size are unaffected. Repository maintenance is smaller, and evaluation claims still require real vendor telemetry.

Verification

  • python scripts/verify_all.py on Windows Python 3.13: 53 tests passed
  • python3 scripts/verify_all.py on WSL Linux Python 3.14: 53 tests passed
  • fresh-checkout simulation without reports/: 53 tests passed
  • skill-creator package validation passed
  • independent diff review found no remaining actionable issues

@dd3ok dd3ok merged commit e47f735 into main Jul 10, 2026
4 checks passed
@dd3ok dd3ok deleted the codex/remove-skill-eval-harness branch July 10, 2026 10:01

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes the custom cross-provider evaluation harness (scripts/run_skill_eval.py), its test fixture (tests/fixtures/fake_eval_runner.py), and its unit tests (tests/test_run_skill_eval.py). It updates the documentation in README.md, README.ko.md, AGENTS.md, and CONTRIBUTING.md to remove references to this harness and the reports/ directory, advising users to perform evaluations manually or using official vendor tools instead. Additionally, test files and validation/experiment-fairness.md are updated to align with these simplified guidelines. There are no review comments provided, so I have no feedback to offer.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant