Skip to content

fix(runtime): refuse successful backend claims that diverge from the plan - #1158

Open
doublewhy wants to merge 2 commits into
devfrom
158-backend-claim-boundary
Open

fix(runtime): refuse successful backend claims that diverge from the plan#1158
doublewhy wants to merge 2 commits into
devfrom
158-backend-claim-boundary

Conversation

@doublewhy

Copy link
Copy Markdown

Plain-language summary

  • Context: test(runtime): characterize backend result integrity #1150's falsification matrix probes whether the runtime boundary can refuse dishonest backend realization claims — the credibility core of "the specification is the unit of exchange."
  • Problem: Four of five fabrications were admitted into the authoritative snapshot: an invented resource, a substituted resource type, a mutation reported as no change, and provisioning results filed under another domain. Only "realizes nothing" was refused (by the SEM-218 gate).
  • Fix: A plan-conformance gate that holds every successful provisioning ApplyResult to the submitted plan before admission; all four lies now come back success=False with runtime.backend-contract-invalid and the untouched baseline snapshot.

Issue mapping

Summary

  • New raes_runtime/backend_plan_conformance.py, wired into _post_apply_contract_result after the existing snapshot-contract checks and before realization-authority disclosure. For each entry in a successful result's snapshot:
    • no plan operation at that address and the entry is new or changed → refused (invented resource / undeclared mutation);
    • resource_type differs from the plan operation → refused (substitution);
    • domain differs from the planned resource's domain (default: provisioning) → refused (cross-domain filing);
    • entry changed but its address is not in changed_addresses and the operation is not UNCHANGED → refused (undisclosed mutation).
  • Scope: successful claims only. A failed apply is already non-authoritative, and its partial snapshot remains preserved for forensics — the existing contract pinned by test_provisioning_failure_preserves_provisioner_snapshot_and_skips_runtime_start. DELETE operations, and the evaluation/orchestration/stop/destroy applies (which carry no provisioning plan), are unaffected.
  • The four strict xfails in test_issue_158_backend_falsification_matrix.py flip to hard assertions (refusal code + no lie in the snapshot), and the module docstring now records the closed state.
  • _supplemental_realization_requirements moves to backend_realization_authority.py (next to its context type) so backend_calls.py stays under the 500-line cap.

Compatibility

  • Consumer-visible tightening: a provisioning backend whose successful claim diverges from the submitted plan is now refused instead of admitted. All in-tree backends (reference, stubs, recording, libvirt, OCI) already conform — 7,011 tests pass unchanged.
  • No SDL field, schema, or portable contract changes.

Verification

  • Falsification matrix: 6/6 hard passes (honest control still admitted; all five fabrications refused).
  • nox -s tests: 7,011 passed; branch-aware coverage gate green (the new module is exercised by the matrix plus the honest paths).
  • Ruff format and lint clean; tools/check_repo_policy.py pass.

🤖 Generated with Claude Code

@doublewhy
doublewhy force-pushed the 158-backend-claim-boundary branch from 01b047f to 1a527e9 Compare August 18, 2026 00:32
Issue #158 asks whether backend agnosticism is credible -- whether the
boundary rejects dishonest, incomplete or overbroad backend claims, or
merely validates happy-path stubs. This answers it with executable probes.

Method matters here. Each case is the real `ReferenceProvisioner`
perturbed by exactly one lie, built from the honest instance's driver and
realization envelope, so a rejection is attributable to that lie. An
earlier hand-rolled provisioner was rejected outright for lacking
realization disclosures, which would have made every result unattributable;
the honest baseline is included as the control that makes the rest mean
something.

Of five fabrications, one is refused: a backend that reports success while
realizing nothing. The other four are admitted, and their fabrications
survive into the authoritative snapshot -- a backend can add a resource the
scenario never authored, relabel what kind of resource it realized, mutate
state while reporting no changed addresses, or file provisioning results
under another domain.

Those four are recorded as strict xfails rather than as assertions of
current behaviour, so a fix turns them into unexpected passes and fails the
suite, forcing the marker to be removed instead of letting the gap close
silently or reopen unnoticed.

Scope is stated in the module: these bound `RuntimeManager.apply`.
`run_target_conformance` is not the discriminator because with default
arguments it also fails the honest reference backend, for the reason
already recorded in issue #663.

Part of #158

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@doublewhy
doublewhy force-pushed the 158-backend-falsification-matrix branch from ece01b1 to 8d8f322 Compare August 18, 2026 07:30
…plan

Issue #158's falsification matrix proved the boundary admitted four of
five dishonest realization claims: an invented resource survived into
the authoritative snapshot, a substituted resource type was accepted, a
mutation reported as no change was accepted, and provisioning results
filed under another domain were accepted.

A new plan-conformance gate (raes_runtime/backend_plan_conformance.py)
now holds every successful provisioning ApplyResult to the submitted
plan before it is admitted:

- an entry that appears or changes at an address with no plan operation
  is refused (invented or undeclared mutation)
- an entry whose resource_type differs from its plan operation is
  refused (substitution)
- an entry whose domain differs from the planned resource's domain is
  refused (cross-domain filing)
- an entry that changed without appearing in changed_addresses is
  refused unless its operation is UNCHANGED (undisclosed mutation)

The gate binds successful claims only: a failed apply is already
non-authoritative and its partial snapshot stays preserved for
forensics (the existing contract exercised by
test_provisioning_failure_preserves_provisioner_snapshot_and_skips_runtime_start).
Evaluation, orchestration, stop, and destroy applies carry no
provisioning plan and are unaffected.

The four strict xfails in the falsification matrix flip to hard
refusal assertions, and _supplemental_realization_requirements moves
to backend_realization_authority.py (with its context type) to keep
backend_calls.py under the file cap.

Closes #158

Verification: the falsification matrix passes as six hard assertions;
nox -s tests green (7,011 tests, coverage gate green); ruff clean;
check_repo_policy pass.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@doublewhy
doublewhy force-pushed the 158-backend-claim-boundary branch from 1a527e9 to 59723bb Compare August 18, 2026 07:31
@Brad-Edwards
Brad-Edwards force-pushed the 158-backend-falsification-matrix branch from 8d8f322 to 677857d Compare August 19, 2026 04:03
Base automatically changed from 158-backend-falsification-matrix to dev August 19, 2026 04:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant