RFC 008 slice 0: validation contracts as executable artifacts by zkwentz · Pull Request #1042 · huggingface/OpenEnv

zkwentz · 2026-08-04T18:33:22Z

Checkpoint

Stacked on #1041 (RFC). Per the RFC's delivery rule, this PR leads with its checkpoint:

PYTHONPATH=src:envs uv run pytest tests/test_validation/ -v   # 59 passed
python scripts/sync_validation_schemas.py --check              # schemas in sync

Every contract in this PR is exercised by a test the day it lands — the "mock API" exists before any implementation.

What this adds (contracts only — no runner, no CLI change, no real grader)

src/openenv/validation/types.py — Level, Lane, CheckStatus, Severity, Verdict, SignatureKind, ProviderCapability. The type split carries the core invariant: graders emit CheckStatus; only the severity policy assigns Severity.
manifest.py — NormalizedManifest + component models, the entire interface between a package format and every grader. Schema rules enforced: judge pin + variance tolerance iff llm_judged; set_state required for injected-state oracles; verifier entry iff kind == "script"; a missing oracle is a valid manifest (graded FAIL later, not a parse error).
signature.py — well-known-file detection table, SignatureError / UnsupportedPackageError, and the unsupported-categories list (exit-code-2 contract).
parsers/, providers/, graders/ — protocols plus working ParserRegistry and GraderRegistry. Grader selection reads the manifest only (a conformance test proves two manifests differing only by signature select identically). Third-party graders register via the openenv.validation.graders entry-point group — zero core changes.
report.py + policy.py + policies/severity-v1.json — report models and the versioned severity policy covering all 45 check ids (33 local + 12 reserved hub/statistical). Hub-lane entries are filtered out entirely for local runs; ERROR results fail closed; unknown ids are an internal error.
schemas/*.schema.json — committed JSON Schema exports of the pydantic models, CI-synced via scripts/sync_validation_schemas.py (same pattern as sync_env_docs.py).
tests/fixtures/validation/ — ten golden/defect packages (served_min_pass, broken_manifest, empty_solution_max_reward, no_oracle, leaky_observation, nondeterministic, leaky_egress, unpinned_judge, harbor_task_min, posttrain_task_min). Each ships its well-known file plus a normalized_manifest.json that doubles as the golden parse result for the parser slices.
tests/test_validation/ — 59 tests: schema round-trips per fixture, policy completeness against the RFC §9 table, protocol conformance against test-only fakes, schema-sync, and the seed of test_checkpoints.py (checkpoints accrete there so regressions are structural).

Also deletes stale untracked __pycache__ bytecode that previously occupied src/openenv/validation/.

Flagged judgment calls (not in the approved design docs)

severity-v1.json bounds are placeholders: max_oracle_tolerance 0.1 · min_floor_margin 0.1 · max_variance_tolerance 0.2 · max_episode_timeout_s 3600.
Reserved hub-lane severities taken from the RFC table; cosign kept warn.
A judge pin declared while llm_judged: false is rejected as a contradiction (strict iff).

Refs #778, #898.

🤖 Generated with Claude Code

Note

Low Risk
New package surface and policy JSON only; no changes to existing CLI or runtime paths until later slices wire the runner.

Overview
Introduces openenv.validation as RFC 008 slice 0: contracts and tests only (no openenv validate runner, parsers, or real graders yet).

The PR adds Pydantic models for a NormalizedManifest (reward/oracle/verifier, resources, network, capabilities, type tags) and a ValidationReport, plus protocols and working ParserRegistry / GraderRegistry (grader selection uses manifest capabilities/tags, not package signature). A versioned severity-v1.json maps all 45 check ids (local + reserved hub/statistical) to lane/severity; apply_policy is the sole path that turns grader CheckStatus into run Verdict (ERROR fails closed; local lane hides hub-only ids).

Shipped artifacts include committed JSON Schemas (packaged via pyproject.toml, kept in sync with scripts/sync_validation_schemas.py), ten validation fixture packages with golden normalized_manifest.json, and 59 tests (fixtures, policy table, protocol fakes, schema sync, slice-0 checkpoint).

^{Reviewed by Cursor Bugbot for commit 51c6d01. Bugbot is set up for automated code reviews on this repo. Configure here.}

bot-ci-comment · 2026-08-04T18:35:24Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cursor · 2026-08-04T18:36:44Z

+            grader = ep.load()
+            if callable(grader) and not isinstance(grader, Grader):
+                grader = grader()
+            self.register(grader)


Entry-point graders never instantiate

Medium Severity

load_entry_points only calls a loaded object when it is callable and not a Grader. A class that declares the protocol fields as class attributes satisfies @runtime_checkable isinstance checks, so it is registered as the class itself. Later applies_to / run calls then bind the first argument as self and break third-party grader loading.

^{Reviewed by Cursor Bugbot for commit 45a86eb. Configure here.}

+
+    signature: SignatureKind
+
+    def parse(self, package_root: Path) -> NormalizedManifest: ...


+    requires_provider: frozenset[ProviderCapability]
+    depends_on: tuple[str, ...]
+
+    def applies_to(self, manifest: NormalizedManifest) -> bool: ...


+
+    def applies_to(self, manifest: NormalizedManifest) -> bool: ...
+
+    def run(self, subject: Subject) -> CheckResult: ...


+
+    base_url: str
+
+    def exec(self, argv: list[str], timeout_s: float) -> ExecResult: ...


+
+    def exec(self, argv: list[str], timeout_s: float) -> ExecResult: ...
+
+    def stop(self) -> None: ...


+        *,
+        deny_egress: bool = True,
+        env_vars: dict[str, str] | None = None,
+    ) -> RunningSubject: ...


cursor · 2026-08-04T18:49:10Z

+
+    model_config = ConfigDict(extra="forbid")
+
+    report_schema_version: str


Minor consistency nit (non-blocking): manifest_schema_version is typed Literal["1"] in manifest.py, so an unknown manifest version is rejected on read. Here report_schema_version is a free str, so a report claiming any version validates. If report consumers are expected to gate on the version too, Literal["1"] would be symmetric; otherwise a one-line note on why reports intentionally accept any version string would clarify intent.

cursor · 2026-08-04T18:49:10Z

+        loaded = 0
+        for ep in entry_points(group=ENTRY_POINT_GROUP):
+            grader = ep.load()
+            if callable(grader) and not isinstance(grader, Grader):


Minor robustness (non-blocking): isinstance(grader, Grader) uses a runtime_checkable Protocol, which checks only attribute presence, not that grader is an instance vs a class. A grader authored as a class with class-level check_id/level/… could pass this check and get registered un-instantiated (or the reverse). Since third-party graders load here via entry points and there's no test covering load_entry_points, isinstance(grader, type) is a more direct "is this a class?" test, and a small entry-point-loading test in a later slice would lock the behavior down.

cursor · 2026-08-04T18:49:10Z

+            if grader.level > max_level:
+                continue
+            if not all(
+                getattr(manifest.capabilities, field)


Minor hardening (non-blocking): if a third-party grader declares a requires_capabilities field name that isn't on CapabilitiesSpec, getattr(manifest.capabilities, field) raises a bare AttributeError here rather than a clear "unknown capability field" error. Given RFC 008's "third parties add graders with zero core changes" goal, validating the field names (or getattr(..., field, <sentinel>) with an explicit error naming the offending field/check id) would fail more helpfully.

First half of the slice-0 contracts: core enums (Level, Lane, CheckStatus, Severity, Verdict, SignatureKind, ProviderCapability), the NormalizedManifest pydantic models with all schema rules (judge pin iff llm_judged, set_state required for injected-state oracles, verifier entry iff script, NetworkPolicy per the Harbor task.toml 1.4 precedent, GPU resource declarations), the committed manifest JSON Schema with its CI sync script, the ten golden/defect fixture packages whose normalized_manifest.json doubles as parser golden output, and the schema round-trip tests. Registries, report/policy contracts, and conformance tests follow in slice 0b. Checkpoint: PYTHONPATH=src:envs pytest tests/test_validation/ -v # 26 passed python scripts/sync_validation_schemas.py --check Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

… 0b (#778, #898) Second half of the slice-0 contracts: signature detection rules and the unsupported-categories error contract; Parser/Provider/Grader protocols with working Parser and Grader registries (third-party graders via the openenv.validation.graders entry-point group); CheckResult/ValidationReport models with the committed report JSON Schema; the versioned severity policy (severity-v1.json, all 45 check ids including reserved hub/statistical ids) with lane filtering and fail-closed verdict application; and the conformance tests: policy completeness against the RFC table, protocol conformance against test-only fakes, report round-trips, and the accreting checkpoint suite. No runner, no CLI change, no real grader — those land per vertical slice. Checkpoint: PYTHONPATH=src:envs pytest tests/test_validation/ -v # 59 passed python scripts/sync_validation_schemas.py --check Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

zkwentz · 2026-08-04T19:11:12Z

Split in two for reviewability: #1044 (slice 0a: core types + normalized manifest + fixtures) and a follow-up slice 0b PR (registries, report schema, severity policy) stacked on #1044. Same content, same branch tip — nothing was dropped.

cursor

Cursor Bugbot has reviewed your changes using default effort and found 2 potential issues.

There are 3 total unresolved issues (including 1 from previous review).

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 51c6d01. Configure here.}

cursor · 2026-08-04T19:14:13Z

+            raise ValueError(
+                "verifier.entry is only valid when verifier.kind is 'script'"
+            )
+        return self


Empty script verifier entry accepted

Medium Severity

VerifierBinding treats script entry as required only when it is None, so an empty string passes validation. That leaves a script verifier with no usable path in a supposedly valid manifest, and later oracle/verifier graders would mis-handle it as a present binding.

^{Reviewed by Cursor Bugbot for commit 51c6d01. Configure here.}

cursor · 2026-08-04T19:14:13Z

+            if entry.severity is Severity.FAIL:
+                return Verdict.FAIL
+            verdict = Verdict.WARN
+    return verdict


Verdict short-circuit hides policy errors

Medium Severity

apply_policy returns immediately on ERROR or fail-severity FAIL, so later unknown or out-of-lane check ids never raise PolicyError. That can turn an internal pipeline error (exit 3) into a normal FAIL verdict (exit 1) when both appear in one run.

^{Reviewed by Cursor Bugbot for commit 51c6d01. Configure here.}

This was referenced Aug 4, 2026

RFC: 008 Environment auto validation #778

Open

Spec-agnostic task validation: openenv validate-task for external task.md packages (PostTrain Arena / BenchFlow) #898

Open

cursor Bot reviewed Aug 4, 2026

View reviewed changes

github-code-quality Bot found potential problems Aug 4, 2026

View reviewed changes

burtenshaw added feature size: extra-large Extra-large pull request labels Aug 4, 2026 — with Cursor

cursor Bot reviewed Aug 4, 2026

View reviewed changes

zkwentz force-pushed the rfc-008/pr2-contracts branch from 45a86eb to 772f5c2 Compare August 4, 2026 18:58

zkwentz and others added 2 commits August 4, 2026 15:09

zkwentz force-pushed the rfc-008/pr2-contracts branch from 772f5c2 to 51c6d01 Compare August 4, 2026 19:09

zkwentz mentioned this pull request Aug 4, 2026

RFC 008 slice 0a: core types, normalized manifest, and fixtures #1044

Open

zkwentz closed this Aug 4, 2026

zkwentz mentioned this pull request Aug 4, 2026

RFC 008 slice 0b: registries, report schema, and severity policy #1045

Open

cursor Bot reviewed Aug 4, 2026

View reviewed changes

Navigation Menu

Uh oh!

RFC 008 slice 0: validation contracts as executable artifacts - #1042

RFC 008 slice 0: validation contracts as executable artifacts#1042
zkwentz wants to merge 2 commits into
rfc-008/pr1-rfcfrom
rfc-008/pr2-contracts

zkwentz commented Aug 4, 2026 •

edited by cursor Bot

Loading

Uh oh!

bot-ci-comment Bot commented Aug 4, 2026

Uh oh!

cursor Bot Aug 4, 2026

Uh oh!

cursor Bot Aug 4, 2026

Uh oh!

cursor Bot Aug 4, 2026

Uh oh!

cursor Bot Aug 4, 2026

Uh oh!

zkwentz commented Aug 4, 2026

Uh oh!

cursor Bot left a comment

Uh oh!

cursor Bot Aug 4, 2026

Uh oh!

cursor Bot Aug 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants


		signature: SignatureKind

		def parse(self, package_root: Path) -> NormalizedManifest: ...


		def applies_to(self, manifest: NormalizedManifest) -> bool: ...

		def run(self, subject: Subject) -> CheckResult: ...


		base_url: str

		def exec(self, argv: list[str], timeout_s: float) -> ExecResult: ...


		def exec(self, argv: list[str], timeout_s: float) -> ExecResult: ...

		def stop(self) -> None: ...


		model_config = ConfigDict(extra="forbid")

		report_schema_version: str

Uh oh!

Conversation

zkwentz commented Aug 4, 2026 • edited by cursor Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checkpoint

What this adds (contracts only — no runner, no CLI change, no real grader)

Flagged judgment calls (not in the approved design docs)

Uh oh!

bot-ci-comment Bot commented Aug 4, 2026

Uh oh!

cursor Bot Aug 4, 2026

Choose a reason for hiding this comment

Entry-point graders never instantiate

Uh oh!

cursor Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 4, 2026

Choose a reason for hiding this comment

Uh oh!

zkwentz commented Aug 4, 2026

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

cursor Bot Aug 4, 2026

Choose a reason for hiding this comment

Empty script verifier entry accepted

Uh oh!

cursor Bot Aug 4, 2026

Choose a reason for hiding this comment

Verdict short-circuit hides policy errors

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

zkwentz commented Aug 4, 2026 •

edited by cursor Bot

Loading