Skip to content

Gate the harness on a deterministic eval suite - #183

Merged
adityak74 merged 1 commit into
mainfrom
feat/eval-harness-suite
Sep 6, 2026
Merged

Gate the harness on a deterministic eval suite#183
adityak74 merged 1 commit into
mainfrom
feat/eval-harness-suite

Conversation

@adityak74

Copy link
Copy Markdown
Contributor

What

zorp-eval grows a second half. harness runs the real zorp-agent
binary against a scripted provider on loopback, and a new harness
continuous integration job gates every pull request on it.

A case is a TOML file. It says what the agent is asked, what the
provider replies, how those bytes reach the client, and what must be
true when the run is over.

Why

Everything that gates today stops at the Model trait. agent.rs
drives a scripted model through the run loop about eighty times, which
is the right shape for tool calls, approval, compaction and
termination. What no gate reached is the layer underneath: the HTTP
client, the streaming parser, the retry bound, the read timeout, and
the store on disk once the process is gone.

That is where every expensive bug of the last month lived. The
streaming path ran with no read timeout at all for months and one call
sat on a socket for three hours eighteen minutes. A provider error
delivered inside an HTTP 200 killed nine of nine benchmark trials. A
truncated stream returned Ok and a nine hour run was misread twice.

The other half of zorp-eval, compat, spawns the agent against a
live provider. That answers a question about models and cannot gate a
merge: a recent nine task run lost four tasks to upstream 404s, and a
gate an outage can turn red teaches people to ignore it. This half has
no network, no key and no model.

The stub moves to its own crate

zorp-agent/tests/sse_stub/mod.rs becomes zorp-stub, so the
transport tests and the suite serve the same bytes. Two sockets that
lie about being a provider in two different ways is exactly the drift
the stub was written to avoid. What stays behind in zorp-agent is
stream_with_patience, which calls stream_sse in process and cannot
live in a crate that depends on nothing.

Rules the case format enforces at load

  • An unknown field is an error. A misspelled expectation that is
    quietly dropped is a case that passes having checked nothing.
  • An empty case directory is an error. A suite that exits zero having
    run nothing is the one failure mode worth refusing outright.
  • A transport field the kind does not read is an error. A retry_after
    on an error_in_stream does nothing, and a case that thinks it does
    is not testing what it says.
  • Every inherited ZORP_ variable is cleared before a case sets its
    own. A connection count means nothing if the developer's shell gets
    to pick the retry bound.

The four seed cases

Case What it pins
a tool call writes a file a streamed call parsed off the wire, run in the workspace, fed back for an answer
the transcript is stored in order what survived to disk, with the tool row between the two assistant rows
a rate limit is retried once a 429 retried, proved by counting connections and not by inference
an error inside a 200 stream is retried an upstream failure a status line called fine, retried before any delta

Each was broken on purpose and watched to fail before it was
committed. A case that passes against a mutated expectation is checking
nothing.

Checks

  • cargo fmt --all --check clean
  • cargo test --workspace --exclude zorp-track --locked passes
  • cargo clippy -p zorp-eval -p zorp-stub -p zorp-agent --all-targets --locked clean
  • the suite itself: 4 cases, 4 passed, and 1 exit code when mutated

Follow-on

docs/superpowers/specs/2026-09-05-harness-eval-catalogue.md (#182)
catalogues 38 cases across ten areas and names what is already proved
at a cheaper level, so nobody writes it twice.

https://claude.ai/code/session_01PG2vuux5tMsjnZBLWJzMyp

Everything that gates today stops at the Model trait. agent.rs drives a
scripted model through the run loop about eighty times, and that is the
right shape for tool calls, approval, compaction and termination. What
no gate reached is the layer underneath: the HTTP client, the streaming
parser, the retry bound, the read timeout, and the store on disk once
the process is gone. That is where every expensive bug of the last month
lived. The streaming path ran with no read timeout at all for months. A
provider error inside an HTTP 200 killed nine of nine benchmark trials.
A truncated stream returned Ok and a nine hour run was misread twice.

zorp-eval grows a second half. `harness` runs the real zorp-agent binary
against a scripted provider on loopback, so a case is a fixed
conversation with fixed transport behaviour and the only thing that can
fail it is the code. The other half, compat, needs a live provider and
deliberately gates nothing: a recent nine task run lost four tasks to
upstream 404s, and a gate an outage can turn red teaches people to
ignore it.

The stub moves out of zorp-agent/tests/sse_stub and into its own crate,
so the transport tests and the suite serve the same bytes rather than
drifting into two sockets that lie about being a provider in different
ways.

A case is data and nothing else. An unknown field is an error, because a
misspelled expectation that is quietly dropped is a case that passes
having checked nothing. An empty case directory is an error for the same
reason. A transport field the kind does not read is an error, because a
retry_after on an error_in_stream does nothing and a case that thinks it
does is not testing what it says. Every inherited ZORP_ variable is
cleared before a case sets its own, since a connection count means
nothing if the developer's shell picks the retry bound.

Four seed cases ship: a streamed tool call that writes a file, the
stored transcript replaying with the tool row between the two assistant
rows, a 429 retried exactly once, and an error object inside a 200
stream retried while nothing has reached the caller. Each was broken on
purpose and watched to fail before it was committed. The catalogue of
the rest is docs/superpowers/specs/2026-09-05-harness-eval-catalogue.md.

Claude-Session: https://claude.ai/code/session_01PG2vuux5tMsjnZBLWJzMyp
@adityak74
adityak74 merged commit 3707aae into main Sep 6, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant