Skip to content

fix(review): verify checkout access before publication - #1170

Merged
jesse-merhi merged 14 commits into
mainfrom
jesse/structured-checkout-access
Aug 17, 2026
Merged

fix(review): verify checkout access before publication#1170
jesse-merhi merged 14 commits into
mainfrom
jesse/structured-checkout-access

Conversation

@jesse-merhi

@jesse-merhi jesse-merhi commented Aug 14, 2026

Copy link
Copy Markdown
Member

ClawSweeper could mark a review complete without proving the review process could read the code it was supposed to inspect. That meant a clean-looking result could be published even if it inspected the wrong checkout or no readable checkout at all. This PR makes the review process read a known file from the exact pull-request commit before it starts a new review or reuses an older one. If that check fails, ClawSweeper stops, records why, and retries instead of publishing.

Change breakdown

Part Files +LOC -LOC
Runner and OpenClaw attestation 2 +296 -15
Exact-head admission and checkout 5 +369 -72
Records, retry, and shared contracts 5 +54 -12
Tests and fixtures 15 +1,130 -30
Total 27 +1,849 -129

These are direct-base net changes. Every file appears once and the rows reconcile to the total.

Proof

The same clean committed range and the same test runner were used for both runs. The runner claimed success but returned the wrong hash for a tracked file.

Before — direct base

process_status: 0
model_review_started: true
review_status: complete
local_checkout_access: verified

The old path trusted the runner, started the review, and recorded a complete result.

After — PR

process_status: 1
model_review_started: false
review_status: failed
review_checkout_inspection_failed: true
local_checkout_access: unverified

The new path detects the mismatch before model review and leaves nothing publishable.

Follow-up — unavailable PR head

previous_head_failed_report_written: false
current_head_failed_report_written: true
model_review_started: false
review_status: failed
review_checkout_inspection_failed: true

The first fresh review found that an unavailable PR head stopped the whole command before recording a retry. The current head now writes the failed record without starting model review.

Reproduce

The comparisons ran on Node 24 with bash .artifacts/pr1170-ab-proof.sh and bash .artifacts/pr1170-missing-head-proof.sh.

Current-head receipt

Head 153a52f231c9b22421336662d3e8e98244fe49f0 passed the real CLI regression and the full repository check. Crabbox receipt: provider aws, lease cbx_f962a3ac5005, machine c7a.8xlarge, exit 0, automatically stopped. The remote scenario used controlled GitHub and runner substitutes; it did not test another provider or a live OpenClaw review.

OpenClaw Bay is unaffected: this changes review admission and retry behavior, not Bay routes, data, or UI.

@clawsweeper

clawsweeper Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

Comment thread src/agent-runner.ts Fixed
Comment thread src/agent-runner.ts Fixed
Co-authored-by: masatohoshino <g515hoshino@gmail.com>
@jesse-merhi
jesse-merhi force-pushed the jesse/structured-checkout-access branch from 7b9ffb2 to add1e6e Compare August 14, 2026 18:18
@clawsweeper clawsweeper Bot added merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. merge-risk: 🚨 security-boundary 🚨 Merging this PR could weaken sandboxing, authorization, credentials, or sensitive data. P2 Normal priority bug or improvement with limited blast radius. rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Aug 14, 2026
@clawsweeper

clawsweeper Bot commented Aug 14, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed August 17, 2026, 1:57 AM ET / 05:57 UTC.

ClawSweeper review

What this changes

This PR materializes the exact pull-request checkout and requires runner-owned read access verification before a review or cached result can be published.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

Keep open for maintainer review. The patch appears correct and has current-head before/after proof, but it deliberately trades a possible false-negative runner preflight for deferred review publication; the PR author is a repository member, so this policy choice should not be auto-closed.

Priority: P1
Reviewed head: 153a52f231c9b22421336662d3e8e98244fe49f0
Owner decision: Required. See Decision needed.

Review scores

Measure Result What it means
Overall readiness 🦞 diamond lobster (5/6) Strong focused proof and extensive regression coverage support the patch; only the maintainer policy choice remains.
Proof confidence 🦞 diamond lobster (5/6) Sufficient (terminal): The PR body provides current-head before/after terminal evidence for the real review command path and records the controlled-run limits and Crabbox receipt.
Patch quality 🦞 diamond lobster (5/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Verified Sufficient (terminal): The PR body provides current-head before/after terminal evidence for the real review command path and records the controlled-run limits and Crabbox receipt.
Evidence reviewed 6 items Current-main gap: The current main runner dispatches the selected agent directly, and a current-tree search found no checkout-inspection or runner-preflight implementation; this PR remains necessary.
Runner attestation: The branch selects a tracked file and verifies the Codex sandbox's hash, or requires OpenClaw to read an exact tracked text path before allowing a verified result.
Exact-head gate: The workflow materializes the reported pull-request head into a detached review worktree, runs the preflight before cache reuse, and throws a retryable checkout-inspection failure before model review if the head is unavailable.
Findings None None.
Security None None.

How this fits together

ClawSweeper's review lane gathers pull-request context, runs an agent against a restricted checkout, and publishes a durable review record. This change inserts exact-head checkout materialization and a read-only runner preflight before model execution or cache reuse.

flowchart TD
  A[Pull request head and context] --> B[Restricted review checkout]
  B --> C[Read-only runner preflight]
  C -->|verified| D[Model review or cache reuse]
  C -->|failed| E[Failed retry record]
  D --> F[Published review record]
Loading

Decision needed

Question Recommendation
Should ClawSweeper require this checkout preflight even when a false-negative result can defer an otherwise valid review? Accept verified-publication gate: Merge the preflight policy so unreadable or mismatched checkouts never produce publishable reviews, with retries handling transient failures.

Why: The patch's integrity benefit is clear, but the acceptable availability cost is an automation-policy choice rather than a mechanical correctness question.

Before merge

  • Resolve merge risk (P1) - A transient or false-negative checkout preflight will defer an otherwise valid review until retry, so merging intentionally favors publication integrity over review availability.
Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Patch scope 27 files, +1,849 / -129 lines The review gate spans runner, checkout, record, retry, and cache paths, so the policy decision affects several publication paths.
Production and test delta production +716 / -99; tests +1,133 / -30 Most growth is focused regression coverage for exact-head materialization, attestation, and retry behavior.

Merge-risk options

Maintainer options:

  1. Accept the bounded retry tradeoff (recommended)
    Merge with the documented policy that a failed preflight records a retryable review failure rather than publishing an unverified result.
  2. Pause for alternative recovery semantics
    Hold the PR if maintainers want a different response to transient preflight failures before changing review publication behavior.

Technical review

Best possible solution:

Make runner-owned exact-head checkout attestation the publication prerequisite while preserving the failed-record retry path for unavailable checkouts.

Do we have a high-confidence way to reproduce the issue?

Yes. The PR documents a controlled Node 24 current-head CLI regression in which a wrong tracked-file hash previously allowed a completed review, while the branch blocks model start and records a retryable failure.

Is this the best way to solve the issue?

Unclear. Exact-head materialization and runner-owned attestation are a narrow, tested solution, but maintainers must explicitly accept the deliberate safety-over-availability policy.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 682ef3ab4abc.

Labels

Label changes:

  • add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.

Label justifications:

  • P1: Publishing reviews for unread or wrong checkouts is an urgent review-automation integrity failure affecting repository triage.
  • merge-risk: 🚨 automation: The patch changes review admission, cache reuse, failure recording, and retry behavior.
  • merge-risk: 🚨 availability: A failed runner preflight intentionally defers review completion until retry.
  • rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The PR body provides current-head before/after terminal evidence for the real review command path and records the controlled-run limits and Crabbox receipt.
  • proof: sufficient: Contributor real behavior proof is sufficient. The PR body provides current-head before/after terminal evidence for the real review command path and records the controlled-run limits and Crabbox receipt.

Evidence

What I checked:

  • Current-main gap: The current main runner dispatches the selected agent directly, and a current-tree search found no checkout-inspection or runner-preflight implementation; this PR remains necessary. (src/agent-runner.ts:34, 682ef3ab4abc)
  • Runner attestation: The branch selects a tracked file and verifies the Codex sandbox's hash, or requires OpenClaw to read an exact tracked text path before allowing a verified result. (src/agent-runner.ts:80, 153a52f231c9)
  • Exact-head gate: The workflow materializes the reported pull-request head into a detached review worktree, runs the preflight before cache reuse, and throws a retryable checkout-inspection failure before model review if the head is unavailable. (src/clawsweeper-review-command-workflow.ts:383, 153a52f231c9)
  • Focused regression coverage: The branch tests that a restricted review materializes the exact head while leaving the base checkout clean, including a force-push mismatch case. (test/review-blob-hydration.test.ts:153, 153a52f231c9)
  • Current-head behavior proof: The PR body records a Node 24 controlled CLI scenario in which the base published after a wrong tracked-file hash, while the branch prevents model start and records a failed retry; it also records Crabbox provider aws and lease cbx_f962a3ac5005. (153a52f231c9)
  • Feature-history routing: Current runner behavior dates to the existing review-automation implementation by Peter Steinberger, making that history the strongest routing signal for the merge-policy decision. (src/agent-runner.ts:80, 583947dc092c)

Likely related people:

  • Peter Steinberger: Current main attributes the existing agent-runner behavior to this review-automation history, including the baseline dispatch path this PR hardens. (role: recent review-automation contributor; confidence: high; commits: 583947dc092c, 96b8703e4c0b; files: src/agent-runner.ts, src/clawsweeper-review-command-workflow.ts)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (42 earlier review cycles; latest 8 shown)
  • reviewed 2026-08-17T04:41:27.115Z sha 80bb018 :: needs changes before merge. :: [P1] Record unavailable review worktrees as retryable failures
  • reviewed 2026-08-17T04:45:36.718Z sha 80bb018 :: needs changes before merge. :: [P1] Record unavailable review worktrees as retryable failures
  • reviewed 2026-08-17T04:58:22.913Z sha 153a52f :: needs maintainer review before merge. :: none
  • reviewed 2026-08-17T05:06:29.659Z sha 153a52f :: needs maintainer review before merge. :: none
  • reviewed 2026-08-17T05:11:26.925Z sha 153a52f :: needs maintainer review before merge. :: none
  • reviewed 2026-08-17T05:16:17.544Z sha 153a52f :: needs maintainer review before merge. :: none
  • reviewed 2026-08-17T05:48:14.802Z sha 153a52f :: needs maintainer review before merge. :: none
  • reviewed 2026-08-17T05:52:29.508Z sha 153a52f :: needs maintainer review before merge. :: none

@clawsweeper clawsweeper Bot added rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Aug 15, 2026
@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added proof: sufficient Contributor real behavior proof is sufficient. and removed merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. labels Aug 15, 2026
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. proof: sufficient Contributor real behavior proof is sufficient. labels Aug 15, 2026
@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 15, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. proof: sufficient Contributor real behavior proof is sufficient. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 15, 2026
@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@jesse-merhi

Copy link
Copy Markdown
Member Author

/clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

@clawsweeper

clawsweeper Bot commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 automation 🚨 Merging this PR could break CI, automerge, proof capture, label sync, or automation. merge-risk: 🚨 availability 🚨 Merging this PR could cause crashes, hangs, restart loops, stalls, or process outages. P1 Urgent regression or broken agent/channel workflow affecting real users now. proof: sufficient Contributor real behavior proof is sufficient. rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants