fix(review): clarify proof scenario coverage - #1286
Merged
Merged
Conversation
Contributor
|
🦞👀 Pull request received. I will update this pull request when review starts. |
clawsweeper
Bot
added
P3
Contributor
|
Codex review: needs maintainer review before merge. Reviewed August 28, 2026, 1:24 PM ET / 17:24 UTC. ClawSweeper reviewWhat this changesThe PR clarifies that a historical Live Verification PASS covers only its declared scenario, adds source-to-scenario proof guidance, and extends the historical publication recipe with scoped classification cases. Merge readinessKeep open: this is a focused, still-unmerged improvement to proof-review guidance and historical PASS rendering; the supplied exact-head production-CLI evidence is sufficient and no introduced correctness defect was found. Priority: P3 Review scores
Verification
How this fits togetherClawSweeper folds historical verification artifacts into durable review reports and renders their results in review comments. Review guidance and proof assessments then determine whether that execution evidence actually covers the changed PR behavior before merge automation can proceed. flowchart LR
A[Historical verification artifact] --> B[Artifact publication command]
C[PR diff and proof assessment] --> D[Review report]
B --> D
D --> E[PASS scope comment]
E --> F[Maintainer merge review]
Before merge
Agent review detailsSecurityNone. Review metrics
Merge-risk optionsMaintainer options:
Technical reviewBest possible solution: Retain the narrow renderer disclaimer and source-aware guidance, with the historical recipe protecting report preservation and repeat-publication behavior. Do we have a high-confidence way to reproduce the issue? Not applicable: this is a review-comment and guidance refinement, and the PR supplies an exact-head production CLI run for the changed rendering path. Is this the best way to solve the issue? Yes: adding the explanation at the common PASS rendering point, paired with focused recipe and unit coverage, is narrower and more maintainable than changing proof-gate semantics. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against ccac077fea2f. LabelsLabel changes:
Label justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Additional instructions
Keep Allow edits from maintainers enabled when opening this PR.
What Problem This Solves
Resolves a presentation and review-guidance gap: historical Live Verification comments show PASS without saying that only the declared scenario and assertions passed. Review guidance also needs an explicit connection from the changed source behavior to the scenario that evidence actually exercises.
This is a focused follow-up to PR #1282 and PR #1283. Those already landed the proof-authority fix and retired-helper cleanup. Main already passes the original 22-case production classification recipe; this PR neither reimplements that fix nor restores the deleted helpers.
Why This Change Was Made
The prompt and two existing schema descriptions now ask reviewers to connect the changed production owner and behavior to the exercised entrypoint, scenario, environment, and observed result or coverage gap. Historical PASS rendering states its limited scope. Relevant changed-help output and substantive terminal evidence remain valid; unrelated help smoke and signing alone do not establish native coverage. Native examples distinguish normal write-half-close success from explicit caller abort, disconnect, or shutdown cancellation. No full-app matrix or video is required.
The recorded semantic assessment remains authoritative. Parser, orchestration, status, label, rating, and publication owners are unchanged, as are receipt identity/plan/assertion validation, independent failure blocking, patch caps, rank-up advice, and existing proof exemptions and gates. No new schema field, version, execution lane, dependency, or budget change is introduced.
User Impact
Maintainers get a clearer explanation of what PASS establishes and a more auditable source-to-scenario proof assessment. Independent sufficient native evidence retains its classification and media attribution even beside an unrelated help PASS. Automatic live proof remains retired.
OpenClaw Bay Impact
No Bay change is needed. There is no observer API, lifecycle, status/telemetry, timing, dashboard data-contract, or action change. Bay remains observer-only; this adds review-comment wording and reviewer guidance only.
Documentation Impact
Reviewed
AGENTS.md,CONTRIBUTING.md,README.md,VISION.md,docs/README.md, and the canonicaldocs/live-proof.mdcompatibility contract. Added incremental scenario guidance beside the existing authority contract without changing it. The active review prompt and schema descriptions are maintained by the review owners; the compatibility document retains its review/publication owners and update triggers.CHANGELOG.mddescribes only this residual guidance and PASS wording.The historical recipe under
docs/proof/historical-proof-classification/retains its original 22 cases and prior repair observations, and adds five explicitly synthetic native/help classification cases. Its README documents their provenance and limits. Verified against baseccac077fea2f63570bd7d91969ee69abcca310e0and committed source206907d968aa9204f50ca4b381b2b18b8f97d046. Update this proof when guidance, rendering, classification, or publication behavior changes.Evidence
The residual preparation passed
pnpm run build:all, all 318 focused tests (zero failures, cancellations, or skips),pnpm run check:static, scoped type-aware source lint, fixture/recipe lint, scoped formatting checks, andgit diff --check. A structural comparison confirms that the schema changes only two descriptions. Protected main owners, test matrices, original proof fixtures, and retired-helper deletions remain unchanged.Focused tests:
Full-suite limitation: an earlier run on the superseded, uncommitted implementation reported 4,011 tests: 4,000 passed, 2 failed, 9 skipped. One failure was an obsolete proof-status expectation; main's superior matrix is preserved here and passes the focused run. The other was the untouched
pnpm validation refreshes the prepared executable before every commandtest, which hitvalidation identity deadline exhausted during raw worktree head. Its subsequent isolated retries each passed that one test, both normally and with coverage instrumentation. Those retries do not establish a clean full-suite result. The hour-long full local coverage run was not repeated for this residual, and no deadline, assertion, or coverage threshold was weakened. Exact-head GitHub CI remains the full gate after PR creation. Fresh isolated Codex precommit review found no actionable findings at the configured P0 threshold; a separate committed-branch review remains pending and is required before merge.A bounded Codex check using the current proof policy and four frozen change/evidence packets produced all four expected classifications: unrelated native/help smoke was insufficient; changed-help evidence, substantive native terminal observations, and independent native evidence alongside help smoke were sufficient. This supports the guidance distinction for those controlled packets only—it is not native execution, authentication of contributor evidence, or a guarantee of future reviewer verdicts.
Real Behavior Proof
Tested committed source head:
206907d968aa9204f50ca4b381b2b18b8f97d046. A fresh maintained production-CLI run on this exact commit passed 27/27 cases with an empty production diff (SHA-256e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855). Its built-owner hashes match the prepared residual, and all 27 classifications and per-case reports, comments, publication results, and repeat-publication receipts match that run. The before/after comparison remains against baseccac077fea2f63570bd7d91969ee69abcca310e0.Claim and surface: the retained production CLI folds historical artifacts, preserves semantic proof and report contents, and renders the added PASS scope disclaimer without changing main's classifications or gates. The exercised path is
live-proof-publish-artifacts→ publication/attachment → report rendering and automation projections. No target command is executed.Command and environment: built with
pnpm run build:all; ran the maintained recipe on unmodified baseccac077fea2f63570bd7d91969ee69abcca310e0, on the prepared residual, and freshly on the exact committed head, each in a new output directory. The committed run reused the verified build of identical source:Local macOS arm64, Node v26.7.0, pinned pnpm 11.10.0; provider
local-process, image and lease not applicable. CLI children receive only PATH, isolated HOME, and LANG. No media manifest exists, so there is no upload; the command folds local artifacts without GitHub comment sync or live-head lookup.Observed results: base
ccac077fea2f63570bd7d91969ee69abcca310e0passed 22/22 original cases; the prepared residual and fresh committed-head run each passed 27/27. Five added cases cover an unrelated help PASS with anapps/macos/Sources/OpenClaw/ExecHostExecutor.swiftcoverage gap, relevant changed-help output, substantive native terminal observations, and independent signed-native linked-artifact/recording attribution. All 22 attached cases preserve frontmatter, the original semantic proof, and unrelated report sections exactly; a second production CLI publication leaves the report byte-identical.Across the original 22 cases, classification/label/rating results, stored reports, and automation markers stayed identical. Eleven PASS comments changed only by adding the sentence below; the other eleven comments stayed byte-identical. Other built owner hashes also remained unchanged.
Before, on already-fixed main:
After:
Artifacts and reproduction: the maintained proof recipe and provenance/limits README are pinned to the tested commit. Local
residual-preparation/main-proof/result.json,residual-preparation/residual-proof/result.json,committed-proof-206907d968aa/result.json, per-case reports/comments/publication results, repeat-publication receipts,committed-before-after.json, andresidual-preparation/rendering-before-after.diffrecord the runs and hashes. The fresh result SHA-256 isc2fb69879808499319d613c522f9cf1e1e5d8e946d3a1a825e76e4a98bfb4964; it records the exact source head and built-owner hashes. These generated artifacts were retained locally, not uploaded; the maintained recipe reproduces the observations. This is a textual rendering observation, not a screenshot or native application recording.Limits: native scenarios and semantic assessments are synthetic classification inputs. This proves actual local publication and rendering behavior, not native correctness or how a model interprets the new guidance. No fresh native/Gateway/terminal-driver execution, live GitHub/R2/queue action, workflow dispatch, target PR action, or deployment was performed. No new automatic proof lane is enabled.