RFC 008: Environment auto-validation - #1041
Conversation
PR1 of the RFC 008 stack. Specifies the six-property quality bar, the four validation levels and their cost budgets, the local/remote boundary (ship-contracts-not-a-hub), the signature -> parser -> normalized manifest -> grader-registry architecture, the oracle/floor semantic contract, the unsupported-package categories, and the full 44-test mapping from #778 to levels, lanes, v1 severities, and stable check ids. Subsumes #898's one-shot task-package lane as parsers under the grader registry. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Alignment Review Report
Two-tier review of the RFC 008 design doc. This PR is documentation-only (rfcs/008-environment-auto-validation.md +463, rfcs/README.md +3), so Tier 1 is essentially empty; the substance is Tier 2 — how the design this RFC proposes lines up with our principles/invariants and with other open RFCs. Overall this is a strong, well-scoped RFC and the "ship contracts, not a hub" framing is sound. The flags below are for discussion, not blockers.
Automated Checks
- Lint: PASS (for this PR). The changed files are Markdown only — zero Python — so the
usort/ruffpipeline oversrc/ tests/ envs/doesn't apply to this diff. Runninglint.shdid surface pre-existing formatting drift in ~25envs/**Python files, but none are touched here. (Heads-up for maintainers, unrelated to this PR:lint.shrunsruff format … envs/, which reformats Python snippets insideenvs/**/README.md, but its revert step only restores*.py, so the hook leaves the tree dirty. Not caused by this PR.) - Debug code: CLEAN (for this PR).
check-debug.shreported only pre-existingprint()/TODOentries insrc/; none are in this PR's files.
Open RFCs Context
- RFC 002 — Env Spec (In Review) — authors @Darktex, @pankit-eng, @jspisak, @zkwentz. Has a pending "Cloud Sandbox Providers" amendment (proposed by @thegovind) that maps hosted sandboxes onto the existing
ContainerProvidercontract. Directly overlaps RFC 008's provider protocol and itsopenenv.yamlmanifest extension. - RFC 004 — Rubrics — RFC 008's
llm_judgedpath reads rubric internals (named_rubrics(),state_dict(), per-child scores) and notes a serialization gap. (Note: RFC 004 has noStatus/Authorsheader block — pre-existing, not part of this PR.) - RFC 001 / 003 (In Review), 005 (In Review), 010 (Draft) — no direct conflicts spotted.
Tier 1: Fixes Required
- None. Docs-only PR with no lint/debug/type/import/security defects in the changed files.
- Optional doc nits (non-blocking):
-
rfcs/008-…md:364— please double-check the severity rollup arithmetic.9 warnand1 advisoryreconcile against the tables, but I couldn't cleanly reproduce30 failgiven the local-vs-reserved split, the#25 → oracle_max + floor_gapsplit, the extrastatic.manifestid, and the split#8sbom/cosign row. Worth a recount (or clarify whether the count is local-lane-only). - Numbering: file is
008(skipping006/007), which matches issue #778's "RFC 008" designation but diverges from README's "next available number" guidance. Fine to keep — just flagging for consistency.
-
Tier 2: Alignment Discussion
Principle Conflicts
None are outright violations — a couple of items to keep honest as the design lands in later PRs:
ALIGNMENT FLAG: set_state / executable-oracle must stay infrastructure-only (never reachable by the agent)
- Principle/Invariant at stake: Agent isolation / Dual API boundary (INVARIANTS.md §Security 1, §Architectural 1)
- The concern: The oracle admits "injected state" via a declared
set_statecapability and a "privileged script executed inside the sandbox." These are new simulation-control-adjacent primitives. The RFC already requires oracle containment (checkruntime.oracle_containment, #22), which is the right instinct — this flag just asks the team to confirm the contract explicitly forbids exposingset_statevia MCP or any agent-facing surface, so it can't become a backdoor to sim control. - Suggested reviewer: @Darktex (author of the agent-isolation / dual-API invariants)
ALIGNMENT FLAG: Secrets must not land in the normalized manifest or report
- Principle/Invariant at stake: No credential exposure (INVARIANTS.md §Security 3)
- The concern: Provider selection is "auto-detected from environment keys (e.g. an HF token)," judge config (model/version/params) is pinned in the manifest, and the report "embeds the normalized manifest verbatim plus a source digest." If tokens/API keys ever flow into the manifest or judge config, they'd be persisted into a machine-readable report that authors share/attach to CI. Please make redaction/secret-exclusion an explicit contract of the manifest+report schemas in PR2.
- Suggested reviewer: @Darktex
Consistency note (not a conflict): the oracle/floor/verifier-sanity checks validate the environment's own reward from outside without computing reward externally — this actively reinforces the "rewards inside environment" invariant. Good.
RFC Conflicts
ALIGNMENT FLAG: Separate validation provider protocol vs. RFC 002's Cloud Sandbox Providers amendment
- RFC at stake: RFC 002 (env-spec, In Review) — pending "Cloud Sandbox Providers" amendment
- The concern: RFC 008 "defines its own provider protocol with declared capabilities and adapts the existing core providers rather than widening the core ABCs," shipping Docker-local + HF Sandbox with egress-denial + in-sandbox-exec. RFC 002's amendment says the opposite by design: hosted sandboxes (incl. HF-style, default-deny egress) become providers by subclassing the existing
ContainerProviderbecause "these are invariants on the existing contract — not new core concepts." Two provider abstractions covering the same runtimes (HF Sandbox) and the same capability (default-deny egress) will drift. The team should decide whether validation extendsContainerProvideror genuinely needs a parallel protocol. - Suggested reviewer: @thegovind (amendment author) + @pankit-eng / @jspisak (RFC 002 authors)
ALIGNMENT FLAG: openenv.yaml validation: block extends the env-spec surface
- RFC at stake: RFC 002 (env-spec, In Review)
- The concern: RFC 008 adds a
validation:section (reward range/tolerances/margins, resources, capabilities, type tags) toopenenv.yamland defines a "normalized manifest schema."openenv.yamlis RFC 002's territory; these fields should be reconciled with the env-spec rather than defined solely in the validation RFC. - Suggested reviewer: @Darktex / @pankit-eng / @jspisak / @zkwentz (RFC 002 authors)
ALIGNMENT FLAG: Redefinition of the existing openenv validate contract (back-compat)
- RFC/at stake: existing
src/openenv/cli/commands/validate.py+_validation.py(the current runtime-API validation contract) - The concern:
openenv validatealready exists and emits apassed/criteria/summaryJSON report, plus a--urlruntime-validation mode; exit code 1 is used today for bad-path/missing-openenv.yaml. RFC 008 redefines the command with a new report schema (verdict/results[]/check_id/status), new flags, and new exit-code semantics (2= unrecognized/unsupported,3= internal). The RFC doesn't say what happens to the existing--urlmode/passedreport or how CI consumers migrate. Please spell out the migration/deprecation path (and whether the old--urlreport is subsumed or removed) in PR2. - Suggested reviewer: @zkwentz (RFC author) / @Darktex
ALIGNMENT FLAG: Cross-RFC coupling to RFC 004 rubric internals + a noted serialization gap
- RFC at stake: RFC 004 (Rubrics)
- The concern: The
llm_judgedpath readsnamed_rubrics()/state_dict()/ per-child scores and states the rubric object "does not serialize model/version/params today," which is why judge pinning is pushed into the manifest. That's a real gap in RFC 004's surface; the two RFCs should agree on whether RFC 004 grows judge-config serialization or whether the manifest is the permanent home. - Suggested reviewer: @Darktex
Process note: the RFC is Status: In Review yet already drives an implementation stack (PR1→PR2 contracts→PR3+ slices), and embeds "the 44-test mapping … Approved 2026-08-04" while the whole RFC is still In Review. Worth confirming the team is comfortable landing contracts/implementation before the RFC reaches Accepted.
Summary
- 0 mechanical issues to fix (docs-only; 2 optional doc nits)
- 2 alignment points for human review (
set_state/oracle containment stays infra-only; secrets excluded from manifest/report) - 4 RFC conflicts to discuss (provider protocol vs RFC 002 amendment;
openenv.yamlextension vs RFC 002; redefinition of existingopenenv validate; RFC 004 rubric coupling + serialization gap)
Suggested reviewers overall: @Darktex (principles/invariants + RFC 002/004), @thegovind (RFC 002 Cloud Sandbox Providers amendment), @pankit-eng / @jspisak (RFC 002 env-spec).
Sent by Cursor Automation: Pre-review
| denial** (subjects start with default-deny networking) and **in-sandbox exec** (oracle scripts, | ||
| containment probes). Validation defines its own provider protocol with declared capabilities and | ||
| adapts the existing core providers rather than widening the core ABCs. v1 ships **Docker-local** | ||
| and **HF Sandbox**. A check whose required provider capability is absent SKIPs with the capability |
There was a problem hiding this comment.
ALIGNMENT FLAG (RFC conflict): This "own provider protocol … adapts the existing core providers rather than widening the core ABCs" is in tension with RFC 002's pending Cloud Sandbox Providers amendment (proposed by @thegovind), which deliberately maps hosted sandboxes (incl. HF-style, default-deny egress) onto the existing ContainerProvider because "these are invariants on the existing contract — not new core concepts." Both cover HF Sandbox and egress-denial. Please reconcile: does validation extend ContainerProvider, or is a parallel protocol truly required? Suggested reviewers: @thegovind, @pankit-eng, @jspisak.
| [--json | --output <path>] | ||
| ``` | ||
|
|
||
| Exit codes: `0` verdict PASS/WARN · `1` verdict FAIL · `2` ambiguous/unrecognized/unsupported |
There was a problem hiding this comment.
ALIGNMENT FLAG (back-compat): openenv validate already exists (cli/commands/validate.py + _validation.py). Today it emits a passed/criteria/summary JSON report and has a --url runtime-validation mode, and uses exit code 1 for bad-path/missing-openenv.yaml. This new contract changes the report schema (verdict/results[]/check_id), adds flags, and reassigns exit codes (2 unrecognized, 3 internal). What happens to the existing --url mode and passed report, and how do current CI consumers migrate? Worth an explicit migration/deprecation note in PR2.
| ```yaml | ||
| # openenv.yaml (validation-relevant fields) | ||
| name: my-swe-env | ||
| validation: |
There was a problem hiding this comment.
ALIGNMENT FLAG (RFC conflict): Adding a validation: block to openenv.yaml extends the env-spec surface owned by RFC 002 (In Review). These fields (reward range/tolerances, resources, capabilities, type tags) should be reconciled with the RFC 002 env-spec rather than defined solely here. Suggested reviewers: RFC 002 authors (@Darktex, @pankit-eng, @jspisak, @zkwentz).
| named. | ||
|
|
||
| Provider selection is auto-detected from environment keys (e.g. an HF token suggests hf-sandbox), | ||
| with a first-run remote-vs-local confirmation, an always-remote flag, and Docker-local as the |
There was a problem hiding this comment.
ALIGNMENT FLAG (No credential exposure — INVARIANTS.md §Security 3): Provider selection auto-detects from environment keys "(e.g. an HF token)," judge config is pinned in the manifest, and the report "embeds the normalized manifest verbatim." Please make it an explicit schema contract that secrets/tokens/API keys are never captured in the manifest or the persisted report (reports get attached to CI and shared). Suggested reviewer: @Darktex.
| **LLM-as-judge is an allowed reward.** Judged environments declare `llm_judged` and pin the judge | ||
| configuration (model, version, params) in the manifest; the oracle check becomes | ||
| `≥ max − declared tolerance` and determinism becomes variance-within-declared-bound rather than | ||
| bit-exact. RFC 004 rubrics are **leveraged, not required**: the contract stays spec-neutral |
There was a problem hiding this comment.
ALIGNMENT FLAG (cross-RFC coupling): The llm_judged path reads RFC 004 rubric internals (named_rubrics(), state_dict(), per-child scores) and notes the rubric object "does not serialize model/version/params today" — hence judge pinning lives in the manifest. That's a real gap in RFC 004's surface. RFC 004 and 008 should agree on whether RFC 004 grows judge-config serialization or the manifest is the permanent home. Suggested reviewer: @Darktex.
| **Rollup:** local `openenv validate` covers 33 of 44 tests (levels 1–3); operators own 9 (all five | ||
| learnable tests, adversarial floor, gameability gap, cross-host reproducibility, immutable | ||
| versioning); 2 are dropped (estargz/nydus conversion — a step too far for v1). v1 severities: | ||
| 30 fail, 9 warn, 1 advisory. |
There was a problem hiding this comment.
Doc nit (non-blocking): please double-check this rollup. 9 warn and 1 advisory reconcile against the tables above, but I couldn't cleanly reproduce 30 fail given (a) the local-vs-reserved split, (b) #25 → semantic.oracle_max + semantic.floor_gap, (c) the extra static.manifest id, and (d) the split #8 sbom/cosign row. Consider clarifying whether 30 fail counts local-lane checks only.
…oviders Two design corrections (Zach, 2026-08-04): (1) network posture follows the Harbor task.toml 1.4 precedent — the manifest declares mode public (default, egress allowed) / no-network / allowlist with allowed_hosts, and runtime.network_policy (was runtime.egress_denied) verifies the sandbox's effective access matches the declaration; verifier hermeticity still runs network-denied. (2) GPU is a provider capability, not an unsupported package category — GPU packages validate on providers that offer GPUs, others SKIP with the capability named. Refs #778. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review finding on #1044: the illustrative openenv.yaml omitted the verifier binding, so the RFC's own example would fail its normative schema. The verifier stays required — how "evaluate this state" is invoked is not defaultable; a served env declares kind: reward_channel explicitly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ign warn Review finding on #1045: the rollup sentence said "30 fail, 9 warn, 1 advisory" but the committed policy carries 34/10/1 over 45 check ids. Both countings are now stated explicitly, and the per-test tally is corrected to 32 fail (the prior 30 was an arithmetic slip — 30+9+1+2 != 44). Test #8's row now pins hub.cosign_signature at warn. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Review direction from Zach on #1045: WELL_KNOWN_FILES must not list formats whose parsers are not implemented in the build. Detection recognizes only supported formats (entries graduate in with their parsers: openenv.yaml at the walking skeleton, task.toml with the Harbor parser, task.md with the PostTrain parser); anything else is refused as unrecognized, exit 2. parser-not-implemented remains as a defensive registry contract only. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
burtenshaw
left a comment
burtenshaw
left a comment
There was a problem hiding this comment.
Reviewed exact head 4c23221; approved for merge.
2 participants
Summary
PR1 of the RFC 008 delivery stack (RFC → contracts → one PR per vertical slice). Adds
rfcs/008-environment-auto-validation.md: the design foropenenv validateand the contracts that let any operator build a validation hub.openenv validateruns levels 1–3 locally; the repo ships contracts (signature rules, parser interface, normalized manifest schema, grader registry, report schema, versioned severity policy, provider interface) — not a hub. Operators own runners, queues, submission APIs, gate policy, and every statistical check.Also updates #778 (drops the reference to the nonexistent adversarial-panel companion spec) and reframes #898 as the PostTrain parser under the grader registry.
Closes nothing; the stack lands the implementation. Next: PR2 (contracts as executable artifacts).
Refs #778, #898.
🤖 Generated with Claude Code
Note
Low Risk
Documentation-only RFC and README index; no runtime, CLI, or schema code changes in this PR.
Overview
Introduces RFC 008: Environment Auto-Validation as PR1 of a planned stack (RFC → contracts → vertical implementation PRs). The RFC defines how local
openenv validateshould grade served environments and one-shot task packages (openenv.yaml, Harbortask.toml, PostTraintask.md) against #778, without shipping hub infrastructure in the repo.The design commits to contracts only in-tree (signature detection, parsers → normalized manifest, grader registry where graders read the manifest not the format, versioned severity policy, JSON report schema, validation provider protocol for Docker-local and HF Sandbox) and explicitly leaves runners, queues, gates, and statistical (level 4) checks to operators. It spells out the semantic contract (oracle required or fail; measured floor gap; determinism as a precondition), CLI flags/exit codes, unsupported package categories, and a full 44-test → check id mapping (33 local, 9 reserved hub/lab, 2 dropped). It also reframes #898 as parsers under the same registry.
rfcs/README.mdgains a Validation section linking the new RFC.Reviewed by Cursor Bugbot for commit 4c23221. Bugbot is set up for automated code reviews on this repo. Configure here.