fix(dashboard): correlate exact-review queue failures

[brokemac79]

Summary

• add one opaque trace identifier across dashboard Worker-to-ExactReviewQueue requests and private failure logs
• classify record-export consistency failures separately from generic handler failures
• route every direct Worker queue.fetch call through the same privacy-bounded wrapper
• preserve all existing public response projections and fail-closed state hydration

Problem

Scheduled state setup can fail with the public, sanitized exact_review_queue_unavailable response. That response is intentionally content-free, but the Worker and Durable Object logs previously had no bounded correlation key and could not distinguish an export consistency failure from a generic handler or platform failure.

The recurrence after #1299 therefore remained ambiguous between record-state inconsistency, a Worker/DO exception, and platform availability or capacity.

Implementation

• generate a UUID v4 per Worker-to-queue request and propagate it in x-clawsweeper-exact-review-trace
• validate the header in the Durable Object before logging it
• reduce request paths to a fixed endpoint allow-list; dynamic record coordinates and query strings are never logged
• retain only fixed transport/category fields, upstream status, and existing platform flags
• classify invalid export watermarks, index/content joins, byte metadata, chunk reconstruction, UTF-8, and canonical record reconstruction as record_export_consistency
• include the observability module in dashboard strict-compilation allow-lists
• retain an opt-in local proof artifact from the existing SQLite-backed Worker/DO harness

No response body, request body, repository/item identity, exception message, filesystem path, stack text, or record content is added to logs.

Validation

Exact base: ce333f9cf9dfb8a29d20ca966b9db4ee78ca1676

Exact head: 279512f1631467955258264cf28e3607845481ca

• pnpm run build:all
• pnpm run check:dashboard-strict
• pnpm run lint:dashboard
• pnpm run check:dashboard-queue-boundary
• node --test test/dashboard-worker-queue-runtime.test.ts — 149/149 passed, including the static bounded-log-call guard that caught and drove the follow-up repair
• node --test test/dashboard-worker-durable-object-error.test.ts test/record-export-bounds.test.ts — 12/12 passed
• node --test --test-concurrency=1 test/dashboard-worker-command-intake.test.ts test/dashboard-worker-webhook-ingress.test.ts test/dashboard-worker-publication-lifecycle.test.ts test/dashboard-worker-queue-policy.test.ts — 112/112 passed
• node --test test/dashboard-worker-queue-runtime.test.ts test/dashboard-worker-bay-records-routes.test.ts test/dashboard-worker-observability.test.ts test/dashboard-worker-status-privacy.test.ts test/dashboard-worker-dashboard-status.test.ts — 367/367 passed after rebasing onto fix(dashboard): make telemetry coverage explicit #1301
• committed Codex review also ran pnpm run test:unit — 314/314 passed
• git diff --check

Rebase integration:

• rebased onto merged fix(dashboard): make telemetry coverage explicit #1301 at ce333f9cf9dfb8a29d20ca966b9db4ee78ca1676
• added the new /telemetry-reconciliation route to the fixed endpoint map as telemetry_reconciliation, including a query-string redaction assertion
• git range-diff 206fe726bbadaae32dcb9822a2511a351a5c9317..181cd6f5c46ac1de48b9da030002e62ebe5782d2 ce333f9cf9dfb8a29d20ca966b9db4ee78ca1676..279512f1631467955258264cf28e3607845481ca shows the three reviewed patches unchanged, plus only the bounded fix(dashboard): make telemetry coverage explicit #1301 integration commit
• the Windows-wide pnpm check formatter gate reports the existing checkout-wide CRLF baseline across 776 files; the scoped formatter, strict compilation, lint, queue-boundary, build, focused tests, integrated tests, and git diff --check pass

Codex review loop:

• dirty production patch: clean after resolving all three accepted P2 findings
• dirty proof-artifact addition: clean
• committed range against exact rebased origin/main after the fix(dashboard): make telemetry coverage explicit #1301 integration: clean; no actionable regressions

Local ClawSweeper review --local-range at the exact head completed with high confidence, found the patch correct, cleared security review, and returned no code comments. Its sole pre-PR gate was attaching an executed Worker-to-Durable-Object trace; that evidence is below.

Real Behavior Proof

Claim: a controlled record-export consistency failure produces the same opaque trace ID at the Durable Object and Worker boundaries, returns only the unchanged public error, and retains no fixture content or coordinates in the logged metadata.

Exercised surface: dashboard Worker route plus SQLite-backed ExactReviewQueue Durable Object using the real request wrapper and record-export store.

Scenario: initialize a signed export, seed a canonical record, remove its required canonical content row, then call the signed /internal/state/records/export Worker route. The same suite also covers invalid byte metadata, malformed base64 chunks, invalid UTF-8, and missing canonical chunks.

Environment and command:

provider=local-container
image=node:24-bookworm
lease=cbx_605b962142a8
run=run_08c6e86888eb
proof_head=181cd6f5c46ac1de48b9da030002e62ebe5782d2
node --test test/dashboard-worker-queue-runtime.test.ts test/dashboard-worker-durable-object-error.test.ts test/record-export-bounds.test.ts

Observed redacted artifact:

{
  "responseStatus": 500,
  "responseBody": { "error": "exact_review_queue_unavailable" },
  "traceId": "<uuid-v4>",
  "durableObject": {
    "phase": "fetch",
    "trace_id": "<same-uuid-v4>",
    "endpoint": "records_export",
    "failure_category": "record_export_consistency",
    "location": null
  },
  "worker": {
    "trace_id": "<same-uuid-v4>",
    "endpoint": "records_export",
    "phase": "request",
    "transport": "throw",
    "upstream_status": null,
    "remote": false,
    "retryable": false,
    "overloaded": false,
    "failure_category": "request_exception"
  },
  "pairedTraceId": true,
  "contentRedacted": true
}

Result: 161/161 focused and queue-runtime tests passed in the container. build:all, strict dashboard compilation, dashboard lint, and the queue-boundary check also passed. The lease auto-stopped after success. Range-diff proves those three container-proven patches are unchanged after the rebase; the fourth commit only adds the #1301 endpoint to the fixed map and is covered by the post-rebase deterministic and integrated tests above.

Limits: this is controlled local-container proof with the in-repository Memory Durable Object and SQLite implementation. It proves response stability, correlation, classification, and redaction for the patch-equivalent runtime changes; a fresh post-rebase container lease was unavailable because the local Docker Desktop Linux daemon was not running. The bounded rebase-only endpoint-map addition is instead covered by deterministic and integrated tests. This proof does not claim production Cloudflare availability, CPU-limit behavior, capacity, or repair of existing production Durable Object state.

Risks and rollout

• the wrapper is shared by Worker-to-queue calls, so the change is intentionally limited to header propagation and private allow-listed logging
• the fix(dashboard): make telemetry coverage explicit #1301 telemetry-reconciliation route is explicitly mapped to a fixed identifier, so its query values cannot fall through into logged endpoint metadata
• OpenClaw Bay is unaffected: this changes no public Bay payload, data source, lifecycle timing or denominator semantics, dashboard rendering, or operator projection; it only adds private Worker/Durable Object failure metadata
• public payloads and status handling are unchanged; scheduled state hydration remains fail-closed
• no retry, notification, queue policy, schema, binding, Wrangler, or Cloudflare configuration changes
• no deployment is part of this PR

The new metadata should distinguish a consistency failure from a structured 5xx, malformed 5xx, remote exception, retryable platform exception, or overload without exposing request content.

Links

• follow-up to fix(state): bound record export candidates #1299

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[brokemac79]

@clawsweeper review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/33388504641
• Updated: 2026-08-31T11:46:21.459Z

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 31, 2026, 10:37 AM ET / 14:37 UTC.

ClawSweeper review

What this changes

Adds an opaque trace ID and fixed failure categories to private dashboard Worker and Exact Review Queue logs while retaining sanitized public queue-error responses.

Merge readiness

⚠️ Ready for maintainer review - 2 items remain

Keep open for normal maintainer merge review. The introduced patch is a bounded private-observability change with no supported correctness or security finding, and its supplied Worker-to-Durable-Object proof covers the central failure-correlation behavior.

Priority: P2
Reviewed head: 279512f1631467955258264cf28e3607845481ca

Review scores

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	Strong focused behavior evidence and redaction coverage support a clean, narrow observability patch.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The supplied local-container trace exercises the real dashboard Worker wrapper and SQLite-backed Durable Object export path: an injected export-consistency fault returns the unchanged opaque error while both private boundaries emit the same redacted trace ID and fixed category. The current-head telemetry-route map addition has deterministic post-rebase coverage; the evidence does not claim Cloudflare production availability.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The supplied local-container trace exercises the real dashboard Worker wrapper and SQLite-backed Durable Object export path: an injected export-consistency fault returns the unchanged opaque error while both private boundaries emit the same redacted trace ID and fixed category. The current-head telemetry-route map addition has deterministic post-rebase coverage; the evidence does not claim Cloudflare production availability.
Evidence reviewed	5 items	Verified introduced surface: The verified merge-base-to-head range changes eight files, adding the shared Worker-to-queue wrapper and bounded observability helper; the working tree remained clean and the range passed whitespace checking. Privacy-bounded failure classification: The Durable Object validates the propagated trace value, reduces paths to a fixed endpoint template, and logs only phase, trace ID, endpoint, failure category, and pre-existing source coordinates. Direct queue calls use one wrapper: The Worker’s direct Exact Review Queue calls now enter the same wrapper, while other queue routes continue through the central request helper; the only remaining direct queue fetch is inside that wrapper.
Findings	None	None.
Security	None	None.

How this fits together

The dashboard Worker sends scheduled and webhook-driven review operations to the Exact Review Queue Durable Object, which persists queue state and exports canonical records. This change correlates failures across that internal boundary without adding record or request content to public responses or logs.

flowchart LR
  A[Scheduled and webhook requests] --> B[Dashboard Worker]
  B --> C[Queue request wrapper]
  C --> D[Exact Review Queue]
  D --> E[Canonical record export]
  C --> F[Bounded Worker failure log]
  D --> G[Bounded queue failure log]
  C --> H[Sanitized public error response]

Loading

Before merge

• Resolve merge risk (P1) - The shared wrapper now observes failures for all Worker-to-Queue calls; the first deployment should confirm private bounded logs and unchanged sanitized responses for platform-generated 5xx cases.
• Complete next step (P2) - No discrete repair is indicated: the current head has focused proof and no actionable review finding, so the remaining action is ordinary maintainer merge review.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test delta	production +259/-36, tests +274/-25	The shared queue-boundary change has slightly more regression and redaction coverage than production code growth.
Affected surface	8 files affected	The change spans the Worker, Durable Object, record-export store, strict compilation lists, and focused tests.

Merge-risk options

Maintainer options:

1. Accept the scoped observability risk (recommended)
   Merge with normal first-deployment observation because the wrapper changes shared internal failure handling but retains the established public error projections.

Technical review

Best possible solution:

Merge the bounded correlation and consistency classification, then observe the first deployment’s private failure metadata to confirm the intended categories without changing public queue behavior.

Do we have a high-confidence way to reproduce the issue?

Yes. The supplied controlled Worker-to-Durable-Object scenario removes canonical content during record export and observes the correlated sanitized failure path; current source also directly establishes the new bounded logging route.

Is this the best way to solve the issue?

Yes. A generated opaque ID plus strict endpoint templates and typed consistency errors is a narrow way to diagnose the ambiguous internal failure without exposing record data or altering public error handling.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against ce333f9cf9df.

Labels

Label justifications:

• P2: This is a bounded operational-observability improvement for an internal review queue, with no evidence of a current user-facing outage.
• merge-risk: 🚨 other: A shared Worker-to-Queue wrapper changes how internal failures are observed across several queue operations, which merits deployment observation beyond passing tests.
• rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The supplied local-container trace exercises the real dashboard Worker wrapper and SQLite-backed Durable Object export path: an injected export-consistency fault returns the unchanged opaque error while both private boundaries emit the same redacted trace ID and fixed category. The current-head telemetry-route map addition has deterministic post-rebase coverage; the evidence does not claim Cloudflare production availability.
• proof: sufficient: Contributor real behavior proof is sufficient. The supplied local-container trace exercises the real dashboard Worker wrapper and SQLite-backed Durable Object export path: an injected export-consistency fault returns the unchanged opaque error while both private boundaries emit the same redacted trace ID and fixed category. The current-head telemetry-route map addition has deterministic post-rebase coverage; the evidence does not claim Cloudflare production availability.

Evidence

What I checked:

• Verified introduced surface: The verified merge-base-to-head range changes eight files, adding the shared Worker-to-queue wrapper and bounded observability helper; the working tree remained clean and the range passed whitespace checking. (dashboard/worker.ts:4808, 279512f16314)
• Privacy-bounded failure classification: The Durable Object validates the propagated trace value, reduces paths to a fixed endpoint template, and logs only phase, trace ID, endpoint, failure category, and pre-existing source coordinates. (dashboard/exact-review-queue.ts:737, 279512f16314)
• Direct queue calls use one wrapper: The Worker’s direct Exact Review Queue calls now enter the same wrapper, while other queue routes continue through the central request helper; the only remaining direct queue fetch is inside that wrapper. (dashboard/worker.ts:4815, 279512f16314)
• Controlled behavior proof supplied: The captured PR body reports a local-container Worker-to-Durable-Object trace with the same UUID at both boundaries, a sanitized 500 response, redaction assertions, and 161 focused passing tests. The three trace-proven commits are reported unchanged by range-diff after rebase; the final endpoint-map addition has focused post-rebase coverage. (test/record-export-bounds.test.ts:529, 279512f16314)
• Relevant merged area history: Recent merged queue-observability work includes failure flags and source-location containment, making its authors useful routing candidates for this shared error boundary. (dashboard/worker.ts:4808, ddc75603add7)

Likely related people:

• Peter Steinberger: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)
• Martin Cleary: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (5 earlier review cycles)

• reviewed 2026-08-31T11:40:10.918Z sha 4fdf0da :: needs maintainer review before merge. :: none
• reviewed 2026-08-31T11:44:43.738Z sha 4fdf0da :: needs maintainer review before merge. :: none
• reviewed 2026-08-31T12:05:30.531Z sha 181cd6f :: needs maintainer review before merge. :: none
• reviewed 2026-08-31T12:13:17.149Z sha 181cd6f :: needs maintainer review before merge. :: none
• reviewed 2026-08-31T14:33:30.332Z sha 279512f :: needs maintainer review before merge. :: none

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/33390154039
• Updated: 2026-08-31T12:07:20.308Z

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/33390814024
• Updated: 2026-08-31T12:15:24.620Z

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

[clawsweeper]

🦞👀
Exact review queued.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/33403923192
• Updated: 2026-08-31T14:39:50.241Z