PR #51 marks exhausted endpoint requests with
[EVALCHEMY_INFRASTRUCTURE_ERROR], but the marker can still reach a benchmark
scorer as model text. Supersedes #50.
Done when:
- Retryable endpoint failures are retried using the same outcome policy as
Harbor.
- An unresolvable failure is retained in samples and run output as
infrastructure state, without contributing a benchmark miss or metric value.
- Regression coverage distinguishes retryable, unresolvable, and genuine empty
model completions.
PR #51 marks exhausted endpoint requests with
[EVALCHEMY_INFRASTRUCTURE_ERROR], but the marker can still reach a benchmarkscorer as model text. Supersedes #50.
Done when:
Harbor.
infrastructure state, without contributing a benchmark miss or metric value.
model completions.