feat: run-evals skill + pin the sim-use binary under test in agent evals#65
Open
onevcat wants to merge 1 commit into
Open
feat: run-evals skill + pin the sim-use binary under test in agent evals#65onevcat wants to merge 1 commit into
onevcat wants to merge 1 commit into
Conversation
Everything in an agent-eval run — device probing, the eval agent's commands, the deterministic verification layer — resolves bare `sim-use` from PATH, so a run could silently evaluate whatever happened to be installed. Make the binary under test explicit and controllable: - run.py gains --sim-use <path>: validates the file, prepends a shim directory (with a `sim-use` symlink) to PATH so every layer of the run resolves the same binary, prints `sim-use under test: <real path> (<version>)` before touching a device, and records the same line in the report header via RunReport meta. - scripts/eval.sh gains -b/--sim-use (env: SIM_USE_BIN) with the same shim applied before its own reachability probing, and its env check now always names the resolved real path + version it will test. - New repo skill .claude/skills/run-evals/ orchestrates the flow for agents and contributors: which-binary confirmation, per-platform environment prep (Device Hub closed on Xcode 27, fixtures installed), cost confirmation, and FAIL triage (skill-prose drift vs CLI regression vs fixture noise). Execution stays delegated to eval.sh/run.py. - e2e/agent-evals/README.md documents the Device Hub prereq and the binary-selection flags; the release skill's eval step references the new skill. Verified: eval.sh syntax + no-cost smoke (-b with a bogus path is rejected; with a real binary the banner prints and the run exits at "0 cases" without spending); run.py override exercised in-process (shim first on PATH, banner resolves the real path, subprocess resolution returns the pinned version). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: onevcat <onevcat@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The agent-eval suite resolves bare
sim-usefrom PATH at every layer (device probing, the eval agent's commands, the verification checks), so a run could silently evaluate whatever binary happened to be installed. This PR makes the binary under test explicit, controllable, and recorded, and packages the whole workflow as a repo skill so future agents and contributors can prepare the environment and run the LLM-driven evals reproducibly.Changes
run.py --sim-use <path>— validates the target, prepends a shim directory (sim-usesymlink) to PATH so every layer of the run resolves the same binary, printssim-use under test: <real path> (<version>)before touching a device, and records the same line in the report header. Default behavior (PATH resolution) is unchanged, but now always announced.scripts/eval.sh -b/--sim-use(env:SIM_USE_BIN) — same shim applied before the wrapper's own reachability probing; the env check now always names the resolved real path + version it will test. Enablesmake eval ARGS="-b .build/out/Products/Debug/sim-use"to eval a development build directly..agents/skills/run-evals/(exposed as/run-evalsvia the.claude/skillssymlink) — orchestration only, execution stays delegated toeval.sh/run.py: which-binary confirmation, per-platform environment prep (Device Hub closed on Xcode 27 — a poisoned boot now fails loudly per fix: guard every HID verb against Device-Hub-poisoned simulators #61 and would red every case; fixture installation checks), cost confirmation rules, and FAIL triage (skill-prose drift vs CLI regression vs fixture noise,fragiletagging).e2e/agent-evals/README.mdgains the Device Hub prereq and the binary-selection section; the release skill's eval step references/run-evals.Verification
bash -non eval.sh;py_compileon run.py.-bwith a bogus path is rejected with a clear error; with a real binary thesim-use under test:banner prints and the run exits at "0 cases selected" without spending.sim-use --versionreturns the pinned binary's version.🤖 Generated with Claude Code