RFC 008 slice 1: walking skeleton — openenv validate end to end

[zkwentz]

Checkpoint

Stacked on #1045 (slice 0b) ← #1044 (slice 0a) ← #1041 (RFC). The full pipeline spine with one real grader — every later slice only adds parsers, graders, or providers to it.

uv run openenv validate envs/echo_env --level static --skip-build
# exit 0, schema-valid report, Verdict: PASS
uv run openenv validate tests/fixtures/validation/broken_manifest --level static --skip-build
# exit 1, FAIL on static.manifest with the schema errors as evidence
uv run openenv validate tests/fixtures/validation/unrecognized_package --level static --skip-build
# exit 2, unrecognized package

All three accreted as checkpoint 1 in tests/test_validation/test_checkpoints.py; checkpoint 0 still green. 102 tests total.

How to review (~700 hand-written lines)

Read in this order:

1. src/openenv/validation/signature.py — detect_signature: exactly one well-known file of a parseable format; the table gains its first entry (openenv.yaml) here, alongside the parser, per the detection direction settled on RFC 008 slice 0b: registries, report schema, and severity policy #1045. Zero matches → unrecognized; two+ → ambiguous (fixture-testable from slice 6; unit-tested against a patched table until then); never a guess.
2. parsers/openenv_yaml.py — pure read (a booby-trapped module in the fixture proves no package code executes); validation: block → NormalizedManifest. Goldens: each fixture's committed normalized_manifest.json must equal the parse result exactly.
3. runner.py — parse → grade → policy → report. A manifest-schema failure is a graded static.manifest FAIL carrying the pydantic errors as evidence and remediation (exit 1), not a crash; grader crashes become ERROR results (fail closed); source_digest = deterministic sha256 of the package tree.
4. graders/static_/manifest.py — the one real grader: declared tolerances vs. the policy's DeclarationBounds.
5. cli/commands/validate.py — the local branch now delegates to the pipeline; exit codes 0/1/2/3; --level/--skip-build/--policy/--json/--output. The legacy --url probe is untouched (folds into runtime.* at slice 3).

Contract amendment to flag

ValidationReport.manifest became NormalizedManifest | None (slice-0b shipped it non-nullable): a package whose declarations fail the manifest schema still gets a schema-valid report — there is simply no valid manifest to embed, and the static.manifest FAIL explains why. Report JSON Schema regenerated.

Also in this diff

• envs/echo_env/openenv.yaml gains a validation: block (reward/resources/capabilities/types) — the first real env under the new manifest, exercised by the checkpoint.
• Fixture openenv.yaml files gain version: 0.1.0 (golden parity) and unrecognized_package/ replaces ambiguous_package/ (ambiguity needs a second parseable format; it returns at slice 6).

Refs #778.

🤖 Generated with Claude Code

---

Note

Medium Risk
Replaces the primary local openenv validate behavior and defines the CLI exit-code contract authors will rely on; scope is validation tooling rather than runtime auth or data paths.

Overview
openenv validate for local packages no longer runs the old multi-mode deployment checks. It now runs the RFC 008 pipeline: detect format via well-known files, parse into a normalized manifest, run graders up to --level, apply severity policy, and emit a human or JSON report with exit codes 0 (pass/warn), 1 (fail), 2 (unrecognized/unsupported), 3 (internal). New flags include --level, --skip-build, --policy, --json, and --output. The --url runtime probe is unchanged.

The spine adds detect_signature (only openenv.yaml in this build), OpenEnvYamlParser (pure read of the validation: block), run_validation (parse → grade → policy, plus deterministic source_digest), and StaticManifestGrader (static.manifest: declared reward tolerances vs policy bounds). Parse/schema failures surface as graded static.manifest FAIL with evidence via ManifestError, not exit 2. ValidationReport.manifest is now optional when the manifest cannot be produced; the report JSON schema was updated accordingly.

envs/echo_env/openenv.yaml gains a validation: block as the first real env under the new manifest. Test fixtures pick up version: 0.1.0; checkpoint and CLI/parser/runner/signature tests lock the slice-1 contract.

^{Reviewed by Cursor Bugbot for commit 1a9d9df. Bugbot is set up for automated code reviews on this repo. Configure here.}

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 3 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 1a9d9df. Configure here.}

[cursor]

Ambiguity test is falsely green

Medium Severity

test_ambiguous_package_raises_signature_error points at a removed ambiguous_package fixture. The nonexistent path raises not a package directory, and match="ambiguous" still passes only because that substring appears in the path name, so the test never exercises real ambiguity detection through run_validation.

 

^{Reviewed by Cursor Bugbot for commit 1a9d9df. Configure here.}

[cursor]

Digest not cross-platform deterministic

Medium Severity

source_digest hashes str(relative_path), which uses backslashes on Windows and forward slashes elsewhere. Nested package files therefore produce different digests for the same tree across OSes, breaking the report's claimed deterministic provenance.

 

^{Reviewed by Cursor Bugbot for commit 1a9d9df. Configure here.}

[cursor]

Validation temp dirs never cleaned

Low Severity

Each successful parse creates an openenv-validate- directory via tempfile.mkdtemp and never removes it after grading. Repeated run_validation calls (CLI, tests, or a hub) accumulate orphaned temp directories under the system temp root.

 

^{Reviewed by Cursor Bugbot for commit 1a9d9df. Configure here.}

[cursor]

Alignment Review Report

Two-tier review of RFC 008 slice 1 (walking skeleton). Reviewed diff e1a5f68...1a9d9df against rfc-008/pr2-contracts.

Automated Checks

• Lint (PR files): PASS — ruff format --check, ruff check, and usort check are all clean on the 15 changed .py files. The repo-wide .claude/hooks/lint.sh reports formatting drift, but every flagged file is under envs/ and untouched by this PR (pre-existing).
• Debug code: CLEAN — no print/breakpoint/TODO introduced in the changed files (check-debug.sh output is entirely pre-existing src/ code).
• Tests: FAIL — full suite: 8 failed, 1624 passed, 133 skipped. All 8 failures are in the pre-existing tests/test_cli/test_validate.py; the 102 new tests/test_validation/ tests all pass.

Open RFCs Context

• RFC 008 — Environment Auto-Validation (In Review, @zkwentz): This PR is an explicit slice of it ("PR3+ — implementations … walking skeleton"). It is well-aligned with the RFC design: signature → parse → grade → severity policy → report; the exit-code contract (0/1/2/3); parser as a pure read; static.manifest as the first grader. This implements the RFC rather than conflicting with it.
• Other RFCs (001 abstractions, 002 env-spec, 003 MCP, 004 rubrics): no conflicts — slice 1 does not touch reward computation, the agent/infra API boundary, or client/server code.

Tier 1: Fixes Required

• tests/test_cli/test_validate.py — 8 tests broken by the local-validation rewrite (CI-blocking). validate.py replaced the old local path (validate_multi_mode_deployment / build_local_validation_json_report, the [OK] output, and the main()-guard + dependency checks) with the RFC 008 pipeline, but the tests asserting the old behavior were not updated or removed. I verified all 12 of these tests pass on the base commit and 8 now fail on HEAD. Failing:

  • test_validate_command_local_path_still_works
  • test_validate_command_local_json_output
  • test_validate_command_rejects_environment_package_as_runtime_dependency
  • test_validate_command_accepts_dockerfile_managed_openenv_runtime
  • test_validate_command_accepts_main_call_with_arguments
  • test_validate_command_rejects_nested_main_guard
  • test_validate_command_accepts_later_top_level_main_guard
  • test_validate_command_syntax_error_fallback_requires_dunder_main

  Fix: delete or rewrite these obsolete tests to match the new pipeline contract. The 3 runtime/--url tests and test_validate_command_rejects_mixed_path_and_url still pass and should stay.

• src/openenv/validation/runner.py:149 — temp-dir leak (minor). tempfile.mkdtemp(...) is created unconditionally per run and never cleaned up; at the static level no grader writes to outputs_dir, so every openenv validate (and each run_validation test) leaves an empty /tmp/openenv-validate-* directory behind. Consider deferring creation until a grader needs it, or cleaning up when empty.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. Reward declaration in the manifest is validation metadata, not external reward computation (rewards stay inside the environment). The parser is a verified pure-read (no package import/exec), consistent with the "Secure" property and client-server separation.

RFC Conflicts

ALIGNMENT FLAG: Old openenv validate structural checks are silently dropped

• Principle/RFC at stake: RFC 008 (In Review) — it redefines openenv validate; principles "Be hands-on" / "minimize lifecycle deltas".
• The concern: The pre-RFC-008 command validated deploy-ability (openenv runtime dependency present, a module-scope main() guard, multi-mode deployment). The new pipeline drops those checks, and none of RFC 008's 44 acceptance tests obviously covers "the server has a runnable entry point." Two consequences worth a conscious decision: (1) the broken tests above are the direct symptom, and (2) 37 of 38 in-repo envs now FAIL static.manifest because they lack a validation: block (only echo_env, updated in this PR, has one). Both are plausible for a walking skeleton, but the team should confirm the old checks are deliberately retired (vs. migrated to a later static grader) and acknowledge the interim behavior change.
• Suggested reviewer: @zkwentz (RFC 008 author / PR author); @Darktex (owner of PRINCIPLES.md & INVARIANTS.md).

Summary

• 2 mechanical issues to fix — 1 CI-blocking (8 broken tests), 1 minor (temp-dir leak).
• 1 alignment point for human review — retirement of the old validate structural checks and the resulting repo-wide interim behavior change.
• 0 hard RFC conflicts — this PR implements RFC 008 rather than conflicting with it.

  

_{Sent by Cursor Automation: Pre-review}

[cursor]

Tier 1 (CI-blocking): this rewrite replaces the old local-validation path (validate_multi_mode_deployment / build_local_validation_json_report, [OK] output, main()-guard + dependency checks) with the RFC 008 pipeline, but the tests that assert the old behavior in tests/test_cli/test_validate.py were not updated. 8 of them now fail on HEAD (all 12 pass on the base commit). Please delete or rewrite the obsolete local-path tests to match the new pipeline contract — the --url/runtime tests and the mixed-path guard test still pass and should be kept.

[cursor]

Tier 1 (minor): tempfile.mkdtemp(...) creates a directory that is never cleaned up. At the static level no grader writes to outputs_dir, so every openenv validate run (and each run_validation test) leaves an empty /tmp/openenv-validate-* dir behind — unbounded accumulation over time. Consider deferring outputs_dir creation until a grader that writes artifacts actually runs, or removing it when empty.

[cursor]

Alignment Review Report

Two-tier review of RFC 008 slice 1 (walking skeleton) — PR3 of the stacked RFC-008 series, base rfc-008/pr2-contracts. Reviewed diff e1a5f68...1a9d9df.

Automated Checks

• Lint (PR files): PASS — ruff format --check, ruff check, and usort check are all clean on the 15 changed .py files. (.claude/hooks/lint.sh exits non-zero, but every file it flags is pre-existing formatting drift under envs/ and untouched by this PR; CI lint scope is src/+tests/ only.)
• Debug code: CLEAN — no print/breakpoint/TODO introduced in changed files (check-debug.sh output is entirely pre-existing src/ code).
• Tests: 8 FAILING (regression) — full local suite: 8 failed, 1624 passed, 133 skipped. All 8 failures are in the pre-existing tests/test_cli/test_validate.py; the 102 new tests/test_validation/ tests all pass.

CI note: test.yml (the pytest + lint suite) only runs on PRs targeting main/release. Because this PR stacks onto rfc-008/pr2-contracts, that workflow does not run here — gh pr checks shows only build_pr_documentation. So the failures below will not turn this PR's checks red; they surface only once the stack retargets/merges to main. Easy to miss — please fix before then.

Open RFCs Context

• RFC 008 — Environment Auto-Validation (In Review, @zkwentz): this PR is an explicit slice of it ("PR3+ — implementations … walking skeleton"). It faithfully implements the RFC design: signature → parse → grade → severity policy → report; the 0/1/2/3 exit-code contract; parser as a verified pure read; static.manifest as the first grader; graders read the manifest, never the signature. No conflict — it implements the RFC.
• Other RFCs (001/002/003/004/005/010): no conflicts — slice 1 doesn't touch reward computation, the agent/infra API boundary, or client/server code.

Tier 1: Fixes Required

• tests/test_cli/test_validate.py — 8 tests broken by the local-validation rewrite (real regression). validate.py replaced the old local path (validate_multi_mode_deployment / build_local_validation_json_report, the [OK] output, main()-guard + dependency checks) with the RFC 008 pipeline, but the tests asserting the old behavior weren't updated or removed. I verified all 12 of these pass on the base commit and 8 now fail on HEAD. Failing:

  • test_validate_command_local_path_still_works
  • test_validate_command_local_json_output
  • test_validate_command_rejects_environment_package_as_runtime_dependency
  • test_validate_command_accepts_dockerfile_managed_openenv_runtime
  • test_validate_command_accepts_main_call_with_arguments
  • test_validate_command_rejects_nested_main_guard
  • test_validate_command_accepts_later_top_level_main_guard
  • test_validate_command_syntax_error_fallback_requires_dunder_main

  Fix: delete or rewrite these obsolete tests to match the new pipeline contract. The 3 runtime/--url tests and test_validate_command_rejects_mixed_path_and_url still pass and should stay.

• src/openenv/validation/runner.py:149 — temp-dir leak (minor). tempfile.mkdtemp(...) is created unconditionally per run and never cleaned up; at the static level no grader writes to outputs_dir, so every openenv validate run (and each run_validation test) leaves an empty /tmp/openenv-validate-* dir behind. Consider deferring creation until a grader that writes artifacts runs, or removing it when empty.

Tier 2: Alignment Discussion

Principle Conflicts

None identified. Reward declaration in the manifest is validation metadata, not external reward computation (rewards stay inside the environment). The parser is a verified pure-read (no package import/exec), consistent with the "Secure" property and client-server separation.

RFC Conflicts

ALIGNMENT FLAG: Old openenv validate structural checks are silently dropped

• Principle/RFC at stake: RFC 008 (In Review) — it redefines openenv validate; principles "Be hands-on" / "minimize lifecycle deltas".
• The concern: the pre-RFC-008 command validated deploy-ability (openenv runtime dependency present, a module-scope main() guard, multi-mode deployment). The new pipeline drops those checks, and none of RFC 008's 44 acceptance tests obviously covers "the server has a runnable entry point." Two consequences worth a conscious decision: (1) the broken tests above are the direct symptom, and (2) 37 of 38 in-repo envs now FAIL static.manifest because they lack a validation: block (only echo_env, updated in this PR, has one). Both are plausible for a walking skeleton, but the team should confirm the old checks are deliberately retired (vs. migrated to a later static grader) and acknowledge the interim behavior change.
• Suggested reviewer: @zkwentz (RFC 008 author / PR author); @Darktex (owner of PRINCIPLES.md & INVARIANTS.md).

Summary

• 2 mechanical issues to fix — 1 real test regression (8 broken tests, not surfaced by this stacked PR's checks but will block at merge-to-main), 1 minor (temp-dir leak).
• 1 alignment point for human review — retirement of the old validate structural checks and the resulting repo-wide interim behavior change.
• 0 hard RFC conflicts — this PR implements RFC 008 rather than conflicting with it.

  

_{Sent by Cursor Automation: Pre-review}

[cursor]

Tier 1 (test regression): this rewrite replaces the old local-validation path (validate_multi_mode_deployment / build_local_validation_json_report, [OK] output, main()-guard + dependency checks) with the RFC 008 pipeline, but the tests asserting the old behavior in tests/test_cli/test_validate.py weren't updated — 8 of them now fail on HEAD (all 12 pass on the base commit). Note test.yml doesn't run on this stacked PR (base is a feature branch), so this won't turn the PR's checks red, but it will block when the stack retargets to main. Please delete or rewrite the obsolete local-path tests to match the new pipeline contract; keep the --url/runtime tests and the mixed-path guard test.

[cursor]

Tier 1 (minor): tempfile.mkdtemp(...) creates a directory that is never cleaned up. At the static level no grader writes to outputs_dir, so every openenv validate run (and each run_validation test) leaves an empty /tmp/openenv-validate-* dir behind — unbounded accumulation over time. Consider deferring outputs_dir creation until a grader that writes artifacts actually runs, or removing it when empty.