Gate the harness on a deterministic eval suite - #183
Merged
Conversation
Everything that gates today stops at the Model trait. agent.rs drives a scripted model through the run loop about eighty times, and that is the right shape for tool calls, approval, compaction and termination. What no gate reached is the layer underneath: the HTTP client, the streaming parser, the retry bound, the read timeout, and the store on disk once the process is gone. That is where every expensive bug of the last month lived. The streaming path ran with no read timeout at all for months. A provider error inside an HTTP 200 killed nine of nine benchmark trials. A truncated stream returned Ok and a nine hour run was misread twice. zorp-eval grows a second half. `harness` runs the real zorp-agent binary against a scripted provider on loopback, so a case is a fixed conversation with fixed transport behaviour and the only thing that can fail it is the code. The other half, compat, needs a live provider and deliberately gates nothing: a recent nine task run lost four tasks to upstream 404s, and a gate an outage can turn red teaches people to ignore it. The stub moves out of zorp-agent/tests/sse_stub and into its own crate, so the transport tests and the suite serve the same bytes rather than drifting into two sockets that lie about being a provider in different ways. A case is data and nothing else. An unknown field is an error, because a misspelled expectation that is quietly dropped is a case that passes having checked nothing. An empty case directory is an error for the same reason. A transport field the kind does not read is an error, because a retry_after on an error_in_stream does nothing and a case that thinks it does is not testing what it says. Every inherited ZORP_ variable is cleared before a case sets its own, since a connection count means nothing if the developer's shell picks the retry bound. Four seed cases ship: a streamed tool call that writes a file, the stored transcript replaying with the tool row between the two assistant rows, a 429 retried exactly once, and an error object inside a 200 stream retried while nothing has reached the caller. Each was broken on purpose and watched to fail before it was committed. The catalogue of the rest is docs/superpowers/specs/2026-09-05-harness-eval-catalogue.md. Claude-Session: https://claude.ai/code/session_01PG2vuux5tMsjnZBLWJzMyp
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
zorp-evalgrows a second half.harnessruns the realzorp-agentbinary against a scripted provider on loopback, and a new
harnesscontinuous integration job gates every pull request on it.
A case is a TOML file. It says what the agent is asked, what the
provider replies, how those bytes reach the client, and what must be
true when the run is over.
Why
Everything that gates today stops at the
Modeltrait.agent.rsdrives a scripted model through the run loop about eighty times, which
is the right shape for tool calls, approval, compaction and
termination. What no gate reached is the layer underneath: the HTTP
client, the streaming parser, the retry bound, the read timeout, and
the store on disk once the process is gone.
That is where every expensive bug of the last month lived. The
streaming path ran with no read timeout at all for months and one call
sat on a socket for three hours eighteen minutes. A provider error
delivered inside an HTTP 200 killed nine of nine benchmark trials. A
truncated stream returned
Okand a nine hour run was misread twice.The other half of
zorp-eval,compat, spawns the agent against alive provider. That answers a question about models and cannot gate a
merge: a recent nine task run lost four tasks to upstream 404s, and a
gate an outage can turn red teaches people to ignore it. This half has
no network, no key and no model.
The stub moves to its own crate
zorp-agent/tests/sse_stub/mod.rsbecomeszorp-stub, so thetransport tests and the suite serve the same bytes. Two sockets that
lie about being a provider in two different ways is exactly the drift
the stub was written to avoid. What stays behind in
zorp-agentisstream_with_patience, which callsstream_ssein process and cannotlive in a crate that depends on nothing.
Rules the case format enforces at load
quietly dropped is a case that passes having checked nothing.
run nothing is the one failure mode worth refusing outright.
retry_afteron an
error_in_streamdoes nothing, and a case that thinks it doesis not testing what it says.
ZORP_variable is cleared before a case sets itsown. A connection count means nothing if the developer's shell gets
to pick the retry bound.
The four seed cases
Each was broken on purpose and watched to fail before it was
committed. A case that passes against a mutated expectation is checking
nothing.
Checks
cargo fmt --all --checkcleancargo test --workspace --exclude zorp-track --lockedpassescargo clippy -p zorp-eval -p zorp-stub -p zorp-agent --all-targets --lockedcleanFollow-on
docs/superpowers/specs/2026-09-05-harness-eval-catalogue.md(#182)catalogues 38 cases across ten areas and names what is already proved
at a cheaper level, so nobody writes it twice.
https://claude.ai/code/session_01PG2vuux5tMsjnZBLWJzMyp