Install vitest before validating when the agent removed it - #195
Merged
Conversation
Validation runs `npx vitest`, and the generated vitest.config.ts imports `vitest/config`, resolved from the workspace. Fixtures supply vitest as a devDependency, but package.json belongs to the agent for the length of a run. An agent scaffolding into an empty directory tends to replace that file rather than edit it, the next install prunes vitest, and npx then fetches it into a cache that does not satisfy the config's import. The run dies at startup with `Cannot find package 'vitest'`. Nothing downstream distinguishes that from a bad result: the eval never executes, and the harness records a failure for work it never graded. A wrong answer and an ungraded one score the same. runValidation now checks for the runner and installs it when missing. The repair must not read as the agent's work, so `--no-save` keeps it out of the manifest and `--no-package-lock` out of the lockfile, which is not gitignored and would otherwise be captured as a file the agent wrote. Verified end to end against a fixture with no vitest: without this the run fails to start, with it the eval executes and grades.
# Conflicts: # packages/agent-eval/src/lib/agents/shared.test.ts
gaojude
marked this pull request as ready for review
August 25, 2026 15:58
devjiwonchoi
approved these changes
Aug 25, 2026
Merged
gaojude
added a commit
to vercel/next-evals-oss
that referenced
this pull request
Aug 25, 2026
Picks up the async judge matchers (vercel-labs/agent-eval#196) and the vitest reinstall guard (vercel-labs/agent-eval#195), moving the harness onto the 2.x line. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An eval whose agent replaces package.json loses vitest, and validation then fails to start rather than grading — so the harness records a failure for work it never ran. This puts the runner back before validating.
Comes up with blank-slate fixtures, where the agent scaffolds a project from scratch and owns the manifest.