Skip to content

fix(live-proof): keep final results visible after cold builds - #1256

Merged
vincentkoc merged 2 commits into
mainfrom
fix/live-proof-output-evidence
Aug 27, 2026
Merged

fix(live-proof): keep final results visible after cold builds#1256
vincentkoc merged 2 commits into
mainfrom
fix/live-proof-output-evidence

Conversation

@vincentkoc

@vincentkoc vincentkoc commented Aug 27, 2026

Copy link
Copy Markdown
Member

What Problem This Solves

Fixes an issue where terminal Live Verification could run the same entry command twice and publish only a cold-build warning prefix, hiding the final meaningful command result in the public review comment.

Observed on openclaw/openclaw#129535 in https://github.com/openclaw/clawsweeper/actions/runs/32900247296.

Why This Change Was Made

Terminal plan parsing now removes exactly one leading run that matches the trimmed automatic entry, while preserving the raw ten-step limit and rejecting plans left with no typed steps. Verification output uses bounded head-and-tail excerpts at both the schema-v1 16K artifact boundary and the 4K public-comment boundary, preserving startup/failure context and the final outcome without filtering dependency warnings.

Browser plans, later repeated commands, distinct commands, sanitization, schema v1, and lifecycle/publication receipts are unchanged.

User Impact

ClawSweeper no longer repeats an identical terminal entry command, and long cold-build output cannot hide the final help text, result marker, or failure diagnostics from the public Live Verification block.

OpenClaw Bay Impact

OpenClaw Bay is unaffected. This change preserves Live Verification schema v1 and does not change lifecycle, publication, receipt, or observer contracts.

Documentation Impact

Updated the active live-proof documentation and review prompt to state that terminal entry executes automatically and must not be repeated as the first run.

Evidence

Pre-fix reproduction

Against origin/main (5cb6777eea2da37e2c2481e15215c2f8be9c774f), the production parser and renderers produced:

{
  "parsedStepCount": 2,
  "artifactChars": 15997,
  "artifactHasHead": true,
  "artifactHasFinal": false,
  "publicHasHead": true,
  "publicHasFinal": false
}

Focused regression proof

Command, Node v26.7.0:

node --test --test-concurrency=1 --test-name-pattern='decision parser removes only|parsed duplicate terminal entry|long terminal failures' test/decision-parser.test.ts test/live-proof.test.ts

Exact result:

✔ decision parser removes only a leading duplicate terminal entry
✔ parsed duplicate terminal entry executes once and preserves long final output publicly
✔ long terminal failures keep command, exit reason, and sanitized tail diagnostics
ℹ tests 3
ℹ pass 3
ℹ fail 0

Additional focused validation:

test/decision-parser.test.ts: 12 passed
test/live-proof.test.ts: 76 passed
Documentation checks passed.
TypeScript build passed.
Formatting, static checks, all three builds, lint, and coverage thresholds passed.

The full local pnpm run check reached 3,761 tests with 3,742 passing and 10 host-fixture failures unrelated to this diff: one existing real-tmux setup failure before target execution and nine repair target-validation failures caused by unavailable local pnpm mirror fixtures or existing hook/timing fixtures. CI remains the authoritative clean-host gate.

Pre-commit and committed-branch Autoreview both reported no actionable findings and patch is correct; the final branch review confidence was 0.98.

Real Behavior Proof

  • Claim: a duplicate terminal entry runs once, and a long cold-build preamble cannot hide the final meaningful result in the public comment.
  • Exercised surface: production parseLiveProofPlan, terminal driveTerminal, schema-v1 artifact construction, encoded report parsing, and public review-comment rendering.
  • Scenario: a deterministic fixture command increments counter.txt, emits more than 20K of build-warning output plus hostile Markdown/HTML marker text, and ends with FINAL_HELP_RESULT; the supplied plan contains the command as both entry and the first run.
  • Command/environment: local Node v26.7.0 and tmux 3.7c, using the production mediaProofCommandRunner and driveTerminal against a private clean-config tmux server. The outer pane's TMUX/TMUX_PANE variables were unset so the real driver received the same default window-0 contract as the hosted environment; no synthetic runner was used.
  • Observable result: counter is 1; artifact output is at most 16K; public output is at most 4K; both retain COLD_BUILD_START and FINAL_HELP_RESULT; hostile output is neutralized; private paths are absent; exactly one truncation marker remains.
  • Artifact/trace: exact focused test output is included above, plus the real-tmux trace below; both are bound to head 5439737c850fab3a507ba849f704e11f4f9aa52e.
  • Limits: deterministic local fixture using the actual production parser/driver/report/comment path; not a hosted review rerun.

Real tmux trace

{
  "head_sha": "5439737c850fab3a507ba849f704e11f4f9aa52e",
  "parser_steps_after_normalization": 1,
  "counter": 1,
  "drive_status": "completed",
  "artifact_chars": 16000,
  "artifact_at_most_16k": true,
  "artifact_has_head": true,
  "artifact_has_final": true,
  "artifact_truncation_markers": 1,
  "public_chars": 4000,
  "public_at_most_4k": true,
  "public_has_head": true,
  "public_has_final": true,
  "public_truncation_markers": 1,
  "public_sanitized_marker": true,
  "private_paths_absent": true,
  "command_identity": "node fixture.mjs"
}

The public excerpt began with COLD_BUILD_START and ended with the sanitized hostile marker followed by FINAL_HELP_RESULT.

Review Notes

Production code: +25/-5 (net +20). Tests: +249/-0. Prompt/docs: +12/-8.

The production growth is the canonical parser normalization plus one reusable output-only truncation path. Generic text truncation and browser behavior remain unchanged.

Related draft #1221 overlaps docs/live-proof.md and test/live-proof.test.ts but does not implement this parser or presentation repair.

Rank-up move disposition

Applied: removed the release-owned CHANGELOG.md entry. The release context remains in this PR body and the squash message.

@clawsweeper

clawsweeper Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

@vincentkoc

Copy link
Copy Markdown
Member Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@vincentkoc

Copy link
Copy Markdown
Member Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@vincentkoc

Copy link
Copy Markdown
Member Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@vincentkoc

Copy link
Copy Markdown
Member Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

🦞👀
Exact review queued.

Re-review progress:

@clawsweeper clawsweeper Bot added P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Aug 27, 2026
@clawsweeper

clawsweeper Bot commented Aug 27, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed August 27, 2026, 12:08 AM ET / 04:08 UTC.

ClawSweeper review

What this changes

The branch removes one leading terminal step that duplicates the automatically executed entry command and preserves both the beginning and final result of long terminal output in verification artifacts and review comments.

Merge readiness

Ready for maintainer review

Keep open for normal maintainer review: this MEMBER-authored draft repairs a current-main live-verification defect with a narrow, tested patch and is not already implemented on main.

Priority: P2
Reviewed head: 5439737c850fab3a507ba849f704e11f4f9aa52e

Review scores

Measure Result What it means
Overall readiness 🐚 platinum hermit (4/6) A focused, well-covered repair with no actionable correctness or security finding in this review.
Proof confidence 🌊 off-meta tidepool Not applicable: Not applicable: the PR author is a MEMBER; the PR body nevertheless supplies a head-bound terminal-driver fixture trace and focused validation results.
Patch quality 🐚 platinum hermit (4/6) No actionable review findings were identified.

Verification

Check Result Evidence
Real behavior Not applicable Not applicable: Not applicable: the PR author is a MEMBER; the PR body nevertheless supplies a head-bound terminal-driver fixture trace and focused validation results.
Evidence reviewed 6 items Current-main behavior remains unfixed: Current main still retains every parsed typed step and applies head-only output trimming, so a leading duplicate command still runs and long final output can be omitted.
Parser repair is narrow: The PR compares normalized terminal commands only at the first typed step, removes that exact duplicate, and still rejects a recommended plan left without typed steps.
Output preservation keeps sanitization: Artifact and public-comment limits now use bounded head-and-tail truncation; untrusted output is still neutralized before comment rendering.
Findings None None.
Security None None.

Live Verification

Command: pnpm run build && node --input-type=module -e "import { createDecisionParser } from './dist/clawsweeper-decision-parser.js'; const parse = createDecisionParser({ isMaintainerAuthorAssociation: () =› false, neutralizeOwnedSectionSpoofing: (v) =› v, sanitizeArchitectureDiagram: (v) =› v }).parseLiveProofPlan; const plan = parse({ status: 'recommended', surface: 'terminal', reason: 'r', payoff: { kind: 'static_text', justification: 'j' }, entry: 'echo once', steps: [{ action: 'run', command: 'echo once' }, { action: 'expect_output', text: 'once' }] }, 'plan'); console.log('normalized steps: ' + plan.steps.length)

Result: FAIL (failed) — step 1 expect_output normalized steps: 1: expected terminal output was not visible within 30 seconds: "normalized steps: 1"

‹private command›: line 1: unexpected EOF while looking for matching ˋ"'

Assertions:

  • FAIL expect_output: normalized steps: 1

How this fits together

Live Verification runs a bounded terminal or browser plan against a reviewed checkout, then stores its result in a durable artifact and renders it into the public review comment. This change sits between plan parsing/output capture and those artifact/comment outputs.

flowchart LR
  A[Review plan] --> B[Plan parser]
  B --> C[Terminal driver]
  C --> D[Verification result]
  D --> E[Durable artifact]
  D --> F[Public review comment]
Loading

Before merge

None.

Agent review details

Security

None.

Review metrics

Metric Value Why it matters
Production versus test delta production +25/-5; tests +249/-0; docs/prompt +12/-8 The small production change is accompanied by focused parser, driver, artifact, and comment-rendering coverage.
Files affected 6 files The patch remains limited to live-proof parsing, rendering, tests, and aligned instructions.

Technical review

Best possible solution:

Retain the parser-bound normalization and head-and-tail rendering approach, then land it after the draft receives normal maintainer review.

Do we have a high-confidence way to reproduce the issue?

Yes—source-reproducible. Current main executes the terminal entry separately from all typed steps and retains only the output prefix at both relevant limits; the PR tests the resulting duplicate-execution and missing-final-output cases.

Is this the best way to solve the issue?

Yes. Normalizing only the leading exact duplicate at the parser boundary and using bounded head-and-tail excerpts at the two output boundaries is the narrowest maintainable repair.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 387cf82c7eba.

Labels

Label changes:

  • add P2: This is a bounded automation correctness repair that prevents misleading or incomplete live-verification evidence without disrupting core runtime availability.
  • add rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • add status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Not applicable: the PR author is a MEMBER; the PR body nevertheless supplies a head-bound terminal-driver fixture trace and focused validation results.

Label justifications:

  • P2: This is a bounded automation correctness repair that prevents misleading or incomplete live-verification evidence without disrupting core runtime availability.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🌊 off-meta tidepool and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: Not applicable: the PR author is a MEMBER; the PR body nevertheless supplies a head-bound terminal-driver fixture trace and focused validation results.

Evidence

What I checked:

  • Current-main behavior remains unfixed: Current main still retains every parsed typed step and applies head-only output trimming, so a leading duplicate command still runs and long final output can be omitted. (src/live-proof/verification.ts:143, 387cf82c7eba)
  • Parser repair is narrow: The PR compares normalized terminal commands only at the first typed step, removes that exact duplicate, and still rejects a recommended plan left without typed steps. (src/clawsweeper-decision-parser.ts:656, 5439737c850f)
  • Output preservation keeps sanitization: Artifact and public-comment limits now use bounded head-and-tail truncation; untrusted output is still neutralized before comment rendering. (src/live-proof/verification.ts:324, 5439737c850f)
  • Focused coverage exercises the affected path: New tests cover exact and trimmed duplicate entries, no-step rejection, the production terminal driver path, final-output retention, and hostile-output neutralization. (test/live-proof.test.ts:1018, 5439737c850f)
  • Live-proof history owner: The current live-proof output path was recently maintained by Peter Steinberger, including the successful-terminal-verification repair. (src/live-proof/verification.ts:143, a958131e8846)
  • Diff scope: The six changed files contain production parser/output changes, targeted tests, and matching active documentation/prompt wording; no dependency, workflow, or lockfile change is present. (src/live-proof/verification.ts:504, 5439737c850f)

Likely related people:

  • Peter Steinberger: Current-main history shows repeated ownership of the live-proof verification path, including the prior terminal-verification repair. (role: recent live-proof feature contributor; confidence: high; commits: a958131e8846, 647503ec44b8, 03c1f452ec49; files: src/live-proof/verification.ts, src/clawsweeper-decision-parser.ts)

Rating scale

Score Internal tier Crab rank Meaning
6/6 S 🦀 challenger crab Exceptional readiness
5/6 A 🦞 diamond lobster Very strong readiness
4/6 B 🐚 platinum hermit Good normal PR; ordinary maintainer review
3/6 C 🦐 gold shrimp Useful, but confidence is limited
2/6 D 🦪 silver shellfish Proof or implementation needs work
1/6 F 🧂 unranked krab Not merge-ready
N/A NA 🌊 off-meta tidepool Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@vincentkoc
vincentkoc marked this pull request as ready for review August 27, 2026 04:31
@vincentkoc
vincentkoc merged commit f211e21 into main Aug 27, 2026
20 of 21 checks passed
@vincentkoc
vincentkoc deleted the fix/live-proof-output-evidence branch August 27, 2026 04:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

P2 Normal priority bug or improvement with limited blast radius. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant