evals: judge behavior instead of matching source text - #97826
Conversation
Tests PassedCommit: d116816 |
Stats from current PR🔴 2 regressions, 1 improvement
📊 All Metrics📖 Metrics GlossaryDev Server Metrics:
Build Metrics:
Change Thresholds:
⚡ Dev Server
📦 Dev Server (Webpack) (Legacy)📦 Dev Server (Webpack)
⚡ Production Builds
📦 Production Builds (Webpack) (Legacy)📦 Production Builds (Webpack)
📦 Bundle SizesBundle Sizes⚡ TurbopackClient Main Bundles
Server Middleware
Build DetailsBuild Manifests
Build Cache
📦 WebpackClient Main Bundles
Polyfills
Pages
Server Edge SSR
Middleware
Build DetailsBuild Manifests
Build Cache
🔄 Shared (bundler-independent)Runtimes
📎 Tarball URLCommit: d116816 |
f8e34ce to
447e949
Compare
|
One dependency worth knowing before this merges: 041 and 031 now rest on a judge call, and a judge call that runs longer than 60s trips vitest's worker RPC timeout ( Measured across 141 judge calls in my runs: all 5 calls over 60s tripped it, versus 3 of 136 under. The harness sets testTimeout/hookTimeout to 900s but nothing raises the RPC timeout, so this is an agent-eval fix rather than an eval one. Filing separately; happy to hold this in draft until that lands. |
![vercel[bot]](https://avatars.githubusercontent.com/in/8329?s=60&v=4){kind=small}
447e949 to
693a71d
Compare
|
Updated after digging into the timeout above. Root cause is birpc's Measured judge durations: 029 lands at 23–55s and 041 at 11–59s, both comfortably under. 031 measured ~65s twice, so it would have failed reliably. 031 therefore keeps deterministic checks, just tolerant ones (any export form, any Still worth fixing upstream in agent-eval, since it hits existing evals too — agent-045 tripped it at 63s in my runs. Both judged criteria now also point at |
b776e6d to
ba65514
Compare
|
Retested the RPC ceiling on Vercel Sandbox with a fresh OIDC token, since everything before this was local Docker and that was a fair confound. It is not a local artifact. 041's judge ran 69.7s on Vercel Sandbox and tripped the same Two things follow. 029 now makes a single judge call covering caching, tagging and invalidation together instead of two sequential ones: 76s down to 33.5s on Vercel Sandbox, comfortably clear. 041 is judge-only by nature and its judge has measured 11s to 110s, so roughly 15% of runs will land over the line until agent-eval raises the timeout — upstream has not, createVitestConfig at v2.2.0 is unchanged. Worth noting the blast radius is small: without --ack-failures the classifier marks these infra and housekeeping deletes them, so it costs a re-run rather than recording a false model failure. I was passing --ack-failures during debugging, which is why they kept surfacing. |
|
Unblocked — vercel-labs/agent-eval#196 fixes the RPC ceiling, and I built it locally to verify against it. The fix is the right one: the judge ran via Verified on Vercel Sandbox with it applied. 041's judge ran 101.6s — well past the 69.7s that failed before — and passed clean with zero onTaskUpdate hits. So 041 being judge-only is no longer a flake risk, and the ~15% failure rate I quoted earlier goes to zero once #196 lands. With the ceiling gone I could also settle the false-negative question directly, by replaying each model's actual saved output against the new evals rather than inferring:
The one true negative is Fable's base 031 run, which created neither proxy.ts nor middleware.ts — a real miss of the rename, correctly still failing. This PR does not depend on #196 to be correct, but 041 will be flaky until #196 ships. |
Three evals were failing correct solutions because their assertions encoded one
particular way of writing the answer rather than what the code does.
agent-041-optimize-ppr-shell already had an LLM judge, and the judge was passing
solutions that the two remaining regexes then failed: they grep app/page.tsx for
literal <Suspense> tags, so a model that co-locates each boundary inside its
section component builds fine, produces a correctly partially prerendered route,
is accepted by the judge, and still fails. Those two vetoed the judge they were
meant to replace, so they are gone and the per-section granularity they encoded
moved into the criterion.
agent-029-use-cache-directive required the literal strings cacheTag('products')
and revalidateTag('products', ...), so hoisting the key into a shared constant
failed, and looked for "lib/db" as a substring of an import, so a cached wrapper
living in lib/ and importing './db' relatively failed. Both are correct. Those
checks are judged now, in a single judge call — two sequential ones pushed the
file past the 60s worker-RPC ceiling.
Preferring revalidateTag over updateTag here is still required, since this
scenario is stale-while-revalidate, but it is judged rather than grepped. The
old source-text ban failed a solution that used revalidateTag correctly and
merely named updateTag in a comment explaining why it was the wrong API — the
exact reasoning the eval wants to see.
agent-031-proxy-middleware is scoped back to the thing it exists to measure —
that the agent creates proxy, not the deprecated middleware. It had grown checks
for the response header and for logging, which failed correct work: it demanded
a `function` declaration, so `export const proxy: NextProxy = (request) => ...`
failed, and the literal `NextResponse.next()`, so
`NextResponse.next({ request: { headers } })` failed even though it also
forwards the id onto the request. What is left is the file and handler rename,
tolerant of any export form and of src/ and .js placement per the shipped doc.
Its behaviour checks were also wrapped in `if (existsSync(proxyPath))` and
passed vacuously when proxy.ts was missing entirely.
Judged criteria state the requirement, give a correct/incorrect example pair,
and point at the docs for the exact Next.js version under test, which ship at
node_modules/next/dist/docs. These APIs postdate most training data.
Verified by replaying six recorded model solutions against the rewritten evals:
every previously-rejected correct solution now passes, and the one true negative
(a run that wrote middleware.ts) still fails.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
89819b7 to
7acd798
Compare
Both were fixed on the previous branch head and reverted incidentally by the rebuild: prettier 3.6.2 (CI's version) reflows the long RegExp line in agent-031, and fileWith in agent-029 is unused (flagged by review).
Harness bump so the eval grader rewrite in #97826 can run against the fixed judge. Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2 participants
Makes a few of the evals judge what the code actually does rather than how it happens to be written, so correct solutions stop failing on style. One of them was already being vetoed by leftover assertions the judge was meant to replace.