fix(runtime): refuse successful backend claims that diverge from the plan - #1158
Open
doublewhy wants to merge 2 commits into
Open
fix(runtime): refuse successful backend claims that diverge from the plan#1158doublewhy wants to merge 2 commits into
doublewhy wants to merge 2 commits into
Conversation
doublewhy
force-pushed
the
158-backend-claim-boundary
branch
from
August 18, 2026 00:32
01b047f to
1a527e9
Compare
Issue #158 asks whether backend agnosticism is credible -- whether the boundary rejects dishonest, incomplete or overbroad backend claims, or merely validates happy-path stubs. This answers it with executable probes. Method matters here. Each case is the real `ReferenceProvisioner` perturbed by exactly one lie, built from the honest instance's driver and realization envelope, so a rejection is attributable to that lie. An earlier hand-rolled provisioner was rejected outright for lacking realization disclosures, which would have made every result unattributable; the honest baseline is included as the control that makes the rest mean something. Of five fabrications, one is refused: a backend that reports success while realizing nothing. The other four are admitted, and their fabrications survive into the authoritative snapshot -- a backend can add a resource the scenario never authored, relabel what kind of resource it realized, mutate state while reporting no changed addresses, or file provisioning results under another domain. Those four are recorded as strict xfails rather than as assertions of current behaviour, so a fix turns them into unexpected passes and fails the suite, forcing the marker to be removed instead of letting the gap close silently or reopen unnoticed. Scope is stated in the module: these bound `RuntimeManager.apply`. `run_target_conformance` is not the discriminator because with default arguments it also fails the honest reference backend, for the reason already recorded in issue #663. Part of #158 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
doublewhy
force-pushed
the
158-backend-falsification-matrix
branch
from
August 18, 2026 07:30
ece01b1 to
8d8f322
Compare
…plan Issue #158's falsification matrix proved the boundary admitted four of five dishonest realization claims: an invented resource survived into the authoritative snapshot, a substituted resource type was accepted, a mutation reported as no change was accepted, and provisioning results filed under another domain were accepted. A new plan-conformance gate (raes_runtime/backend_plan_conformance.py) now holds every successful provisioning ApplyResult to the submitted plan before it is admitted: - an entry that appears or changes at an address with no plan operation is refused (invented or undeclared mutation) - an entry whose resource_type differs from its plan operation is refused (substitution) - an entry whose domain differs from the planned resource's domain is refused (cross-domain filing) - an entry that changed without appearing in changed_addresses is refused unless its operation is UNCHANGED (undisclosed mutation) The gate binds successful claims only: a failed apply is already non-authoritative and its partial snapshot stays preserved for forensics (the existing contract exercised by test_provisioning_failure_preserves_provisioner_snapshot_and_skips_runtime_start). Evaluation, orchestration, stop, and destroy applies carry no provisioning plan and are unaffected. The four strict xfails in the falsification matrix flip to hard refusal assertions, and _supplemental_realization_requirements moves to backend_realization_authority.py (with its context type) to keep backend_calls.py under the file cap. Closes #158 Verification: the falsification matrix passes as six hard assertions; nox -s tests green (7,011 tests, coverage gate green); ruff clean; check_repo_policy pass. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
doublewhy
force-pushed
the
158-backend-claim-boundary
branch
from
August 18, 2026 07:31
1a527e9 to
59723bb
Compare
Brad-Edwards
force-pushed
the
158-backend-falsification-matrix
branch
from
August 19, 2026 04:03
8d8f322 to
677857d
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Plain-language summary
ApplyResultto the submitted plan before admission; all four lies now come backsuccess=Falsewithruntime.backend-contract-invalidand the untouched baseline snapshot.Issue mapping
158-backend-falsification-matrix; the refusal tests are that PR's xfails flipped to hard assertions). Retargets todevwhen test(runtime): characterize backend result integrity #1150 merges; rebase then.Summary
raes_runtime/backend_plan_conformance.py, wired into_post_apply_contract_resultafter the existing snapshot-contract checks and before realization-authority disclosure. For each entry in a successful result's snapshot:resource_typediffers from the plan operation → refused (substitution);domaindiffers from the planned resource's domain (default: provisioning) → refused (cross-domain filing);changed_addressesand the operation is notUNCHANGED→ refused (undisclosed mutation).test_provisioning_failure_preserves_provisioner_snapshot_and_skips_runtime_start.DELETEoperations, and the evaluation/orchestration/stop/destroy applies (which carry no provisioning plan), are unaffected.test_issue_158_backend_falsification_matrix.pyflip to hard assertions (refusal code + no lie in the snapshot), and the module docstring now records the closed state._supplemental_realization_requirementsmoves tobackend_realization_authority.py(next to its context type) sobackend_calls.pystays under the 500-line cap.Compatibility
Verification
nox -s tests: 7,011 passed; branch-aware coverage gate green (the new module is exercised by the matrix plus the honest paths).tools/check_repo_policy.pypass.🤖 Generated with Claude Code