fix(dashboard): make telemetry coverage explicit

[brokemac79]

Summary

• add a signed, read-only, aggregate-only operator reconciliation path that compares public exact-review telemetry with canonical lifecycle projections
• add explicit freshness, coverage, gap, denominator, and nullable-unavailable semantics to the public Bay/Overview contracts
• render zero-attempt or internally inconsistent error-rate telemetry as n/a, not a misleading 0%
• distinguish durable lifecycle availability from independent queue and live activity

Problem

The public Bay and Overview surfaces could make incomplete or unavailable telemetry look authoritative. An empty denominator could render as a successful zero-percent rate, health-history gaps were implicit, and Bay could visually conflate an unavailable durable lifecycle projection with independent queue/live activity.

Operators also lacked a narrow way to validate the published aggregate against canonical lifecycle projections without exporting identity-bearing lifecycle rows.

Implementation

• Add an authenticated operator reconciliation response derived from canonical lifecycle projections and the published aggregate. It returns only bounded counts, nullable duration aggregates, collection state, comparison booleans, and explicit scan limits; it does not return repository names, item numbers, row payloads, workflow details, or run identifiers.
• Bound reconciliation by a 10,000-event limit and a separately reported 100,000-candidate scan limit. Candidate reads use indexed, cursor-paginated pages so a large active population does not cause a false aggregate over_cap result.
• Extend public status and health-history projections with closed freshness and coverage states, explicit expected/observed/usable/failed/missing denominators, largest-gap details, and safe nulls.
• Treat malformed, stale, future-dated, empty, failed, and internally inconsistent observations conservatively rather than synthesizing complete zero-valued telemetry.
• Separate the Durable lifecycle Kanban from the independently sourced Queue and live activity projection, including a correctly labelled main landmark.

Deliberate scope exclusions

This does not move or rebuild the primary duration UI, introduce the proposed 6h/24h/7d hierarchy, collapse diagnostics, claim causality from #1280, publish per-lane timing, add lifecycle-transition persistence, deploy anything, or broaden into a dashboard redesign.

Validation

• exact candidate head: 311e64abb39af2c101835912070abb7824d981b9
• exact base and merge base: 206fe726bbadaae32dcb9822a2511a351a5c9317
• conflict-free rebase from ca58ac2507aa065d447188d9e55cc672ad35f7aa; git range-diff fa535d54a032862b022147a7cba4eaa3bdeb4a7b..ca58ac2507aa065d447188d9e55cc672ad35f7aa origin/main..HEAD reports the patch as equivalent (=)
• node --test test/dashboard-worker-bay-records-routes.test.ts test/dashboard-worker-observability.test.ts test/dashboard-worker-dashboard-status.test.ts test/dashboard-worker-status-privacy.test.ts — 218 passed, 0 failed on the exact head both natively and inside Docker-backed proof
• pnpm run build:all — passed on the exact head, including inside Docker-backed proof
• pnpm run lint:dashboard — passed
• pnpm run check:dashboard-strict — passed
• pnpm run check:dashboard-queue-boundary — passed
• changed-file formatting — passed
• canonical repository formatting for the previously failing reconciliation guard — passed on the exact head
• git diff --check origin/main...HEAD — passed

The broader native-Windows unit sweep reaches unrelated platform/fixture failures (WSL/bash unavailable, symlink EPERM, CRLF/path assumptions, action-ledger partitions, scanner fixtures, and local-review/live-proof fixtures). A bounded current-main baseline check reproduced the representative action-ledger failures at the exact base; the scoped 218-test CSW-143 suite passed on both the branch and the exact-head Docker proof. No broad-suite pass is claimed.

Review closeout

Codex review

The mandatory review loop completed after every accepted finding and proof-affecting change.

Accepted findings and fixes:

• replaced the initial unbounded reconciliation scan with indexed, bounded cursor pagination and an explicit candidate-scan cap; added a 600-active-candidate regression fixture
• corrected the empty Queue and live activity landmark by moving its label to the main beach landmark
• made zero-denominator rendering require a valid numerator, denominator, and rate consistent with the server calculation; distinct null copy now identifies denominator, numerator, or inconsistent-rate failures
• fail closed when lifecycle recovery is pending instead of comparing a public aggregate with an incomplete canonical projection
• reject contradictory freshness metadata and unavailable-history sentinels that carry non-null denominators or samples
• age cached Overview freshness forward in the browser, preserving stale state while tolerating at most 60 seconds of initial Worker/browser clock skew
• applied the canonical formatter to the reconciliation guard after Linux CI identified the narrower Windows changed-file format check as insufficient; no behavior changed

Final results:

• final dirty-patch review: no actionable correctness issues; the unavailable-contract hardening and client-side freshness aging were assessed as internally consistent
• exact-commit review (codex review --commit HEAD): no findings; affected tests, dashboard build, strict check, and lint passed
• actual-base review (codex review --base origin/main): no findings; no actionable correctness issues in the full range, with dashboard build and unit suite passing

Local ClawSweeper committed-range review

Command:

pnpm run review -- --local-range --target-repo openclaw/clawsweeper --base origin/main

The native-Windows gate completed after #1303 repaired the hardened-Git null-device prerequisite. The first subprocess did not inherit the existing host-owned scanner cache on PATH and failed closed with scanner_unavailable; the completed run prepended only the verified external TruffleHog 3.97.1 directory after checking SHA-256 0C0E99305FF86574F18E0CA27736758F5EC307C005FFC1FD14B6702297EB9088 and the exact version. Scanning was not disabled or bypassed.

• reviewed head: 311e64abb39af2c101835912070abb7824d981b9
• reviewed main: 206fe726bbadaae32dcb9822a2511a351a5c9317
• status: complete; checkout access verified; terminal failure false
• source correctness: no findings; patch assessed correct with confidence 0.88
• security: cleared; no concrete authorization bypass, secret exposure, or lifecycle-identity leak found
• proof: sufficient; the review recognized the retained exact-head Docker trace as exercising the signed boundary and rendering contract
• report: .git/clawsweeper/reviews/local-range-1788177479556-28612/0.md

Real Behavior Proof

Claim

An authenticated read-only path can reconcile the public exact-review aggregate with canonical lifecycle projections without exposing lifecycle identities. Public Bay and Overview responses explicitly report freshness, coverage, gaps, and unavailable states; zero denominators do not become false zero-percent success; durable lifecycle availability remains separate from independent queue/live activity.

Exercised surface

• production dashboard Worker served through Wrangler inside Crabbox's Docker-backed local-container provider and the pinned Playwright image
• authenticated Worker-to-Exact Review Durable Object reconciliation boundary
• canonical lifecycle projection and retained aggregate comparison
• public GET /api/status and GET /api/health-history?range=6h
• rendered Overview and Bay pages in Playwright
• focused deterministic tests for complete, partial, stale, empty, malformed, mismatched, retracted, paginated, and unauthorized cases

Scenario / fixture

The proof used loopback endpoints, synthetic credentials, an empty synthetic Durable Object, and synthetic lifecycle/history inputs. It exercised unsigned and signed reconciliation, a fresh public status snapshot, explicit unavailable history coverage, zero-attempt Overview rendering, and a Bay rendering where an empty durable lifecycle projection remains visibly separate from independent queue/live activity. Focused tests additionally perturb canonical-versus-aggregate inputs and require mismatches to be reported.

Command / environment

• exact candidate: 311e64abb39af2c101835912070abb7824d981b9
• exact base: 206fe726bbadaae32dcb9822a2511a351a5c9317
• Crabbox provider: local-container
• container: d39f7a13642f
• execution image: mcr.microsoft.com/playwright:v1.60.0-noble
• image digest: sha256:9bd26ad900bb5e0f4dee75839e957a89ae89c2b7ab1e76050e559790e946b948
• run: run_45def6a55959
• lease: cbx_2ab29a36a0a1
• slug: tidal-prawn-50a5
• result: exit 0, succeeded, leaseStopped=true
• timing: lease 21.208s; bootstrap 1.233s; sync 7.440s; command 28.281s; total 38.760s; end-to-end 60.917s
• invocation: --provider local-container --local-container-image mcr.microsoft.com/playwright:v1.60.0-noble --no-hydrate --artifact-glob ... --require-artifact ...; Crabbox verified all required proof files before stopping the lease

Readable redacted transcript

POST /internal/exact-review/telemetry-reconciliation (unsigned)
  -> 401 {"error":"invalid_signature"}

POST /internal/exact-review/telemetry-reconciliation (valid synthetic signature)
  -> 200
  version=1
  source=canonical-lifecycle-projection-v1
  scope.repository_count=1
  collection.state=complete
  window.minutes=60
  window.event_limit=10000
  window.candidate_scan_limit=100000
  window.candidates_scanned=0
  comparison.canonical_events=0
  comparison.aggregate_events=0
  comparison.missing_events=0
  comparison.unexpected_events=0
  comparison.mismatched_events=0
  comparison.event_sets_match=true
  comparison.public_snapshot_matches_aggregate=true
  normal_direct.samples=0 average_ms=null median_ms=null
  including_legacy_batch.samples=0 average_ms=null median_ms=null

privacy scan
  -> configured repository names absent
  -> item keys, fence keys, delivery IDs, digests, row payloads and run IDs absent

GET /api/status
  -> 200
  freshness.state=fresh
  freshness.cache_state=miss
  freshness.maximum_age_ms=60000
  bay.metrics_state=warming
  bay.timing_coverage_complete=false

GET /api/health-history?range=6h
  -> 200
  coverage.state=unavailable
  expected_slots=72 observed_slots=0 usable_slots=0 failed_slots=0 missing_slots=72
  coverage_percent=0 largest_gap_slots=72 largest_gap_ms=21600000
  freshness.state=unavailable latest_sample_at=null age_ms=null maximum_age_ms=720000

Playwright Overview
  -> Error Rate: n/a
  -> n/a failed / n/a attempts - denominator unavailable

Playwright Bay
  -> Durable lifecycle Kanban: canonical projection only; empty complete snapshot shown
  -> Queue and live activity: independent current projection shown separately
  -> main beach landmark labelled by queue-live-activity-title

focused test transcript
  -> tests 218; pass 218; fail 0; skipped 0

Artifact inventory

The retained 2,054,583-byte artifact is .crabbox/runs/run_45def6a55959/run_45def6a55959-artifacts.tgz. It contains the readable transcript/summary, sanitized signed and unsigned reconciliation JSON, public status/history JSON, Overview/Bay HTML, focused-test TAP, Wrangler log, and overview.png / bay.png. Crabbox required the summary and both screenshots before declaring success. The archive was checked before extraction for safe paths, and the screenshots were visually inspected against the transcript. The tracked proof contract is docs/proof/csw-143-telemetry-accuracy-foundation/behavior-contract.md.

Limits

This synthetic controlled proof establishes route authentication, response privacy shape, calculation/coverage contracts, and rendering behavior. It does not deploy, inspect or mutate production Durable Objects, validate every historical lifecycle row, establish #1280 causality, persist lane-transition timestamps, or prove per-lane timing.

Risks / rollout

• Contract consumers must treat the nullable and closed-state fields as authoritative. Existing surfaces remain compatible while becoming conservative when data is missing or inconsistent.
• Reconciliation is bounded. More than 10,000 qualifying events or 100,000 recent lifecycle candidates yields an explicit unknown/over-cap result instead of a false match.
• The reconciliation route intentionally remains operator-authenticated and aggregate-only. Expanding it to identities or row export requires a separate privacy/security decision.
• No deployment, merge, automerge, or production mutation is requested by this PR.

Related work

• CSW-137 / CSW-137: investigate Bay review-duration metric #1233 established the durable lifecycle aggregate and its bounded public semantics.
• CSW-139 / CSW-139: investigate Bay lane-duration conversation bubbles #1237 separated direct and legacy publication paths.
• fix: retire automatic live proof #1280 is relevant operational context, but this PR makes no causal performance claim.
• fix(review): support Git null device on Windows #1303 repaired the native-Windows hardened-Git prerequisite and is included in this branch's current-main base.

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[clawsweeper]

Codex review: needs maintainer review before merge. Reviewed August 31, 2026, 8:08 AM ET / 12:08 UTC.

ClawSweeper review

What this changes

The PR adds explicit coverage and freshness contracts to dashboard telemetry, an authenticated aggregate-only reconciliation endpoint, and clearer Bay/Overview degraded-state presentation.

Merge readiness

⚠️ Ready for maintainer review - 1 item remains

The telemetry safeguards are unique to this open branch, with a current-head behavior trace and focused coverage; no blocking correctness or security defect was identified. It should remain open for ordinary maintainer review.

Priority: P2
Reviewed head: 311e64abb39af2c101835912070abb7824d981b9

Review scores

Measure	Result	What it means
Overall readiness	🦞 diamond lobster (5/6)	Strong current-head behavior proof and focused coverage support a clean patch, with ordinary maintainer scope review remaining.
Proof confidence	🦞 diamond lobster (5/6)	Sufficient (terminal): The supplied exact-head Crabbox local-container trace exercises the production Worker through rejected and signed reconciliation requests, public status/history contracts, and rendered Bay/Overview states with observed after-fix results.
Patch quality	🦞 diamond lobster (5/6)	No actionable review findings were identified.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (terminal): The supplied exact-head Crabbox local-container trace exercises the production Worker through rejected and signed reconciliation requests, public status/history contracts, and rendered Bay/Overview states with observed after-fix results.
Evidence reviewed	5 items	Pinned PR range: The verified merge-base-to-head range changes 10 files by 1,347 additions and 30 deletions; current main is the exact base, so the telemetry-contract work is not already implemented there. Authenticated route boundary: The Worker authenticates the operator request, derives scope only from verified public repositories, and forwards it to the Durable Object reconciliation route. Reconciliation safeguards: The implementation bounds candidate scans and timing events, fails closed while lifecycle recovery is pending, and compares aggregate timing facts without returning lifecycle rows.
Findings	None	None.
Security	None	None.

How this fits together

OpenClaw Bay and Overview display public projections emitted by the dashboard Worker and its Exact Review Durable Object. This PR changes how lifecycle and health-history data is validated, reconciled, and rendered for operators.

flowchart LR
  A[Lifecycle events] --> B[Exact Review Durable Object]
  B --> C[Dashboard Worker]
  C --> D[Public status and history APIs]
  C --> E[Signed reconciliation API]
  D --> F[Bay and Overview]
  E --> G[Operator verification]

Loading

Before merge

• Complete next step (P2) - No discrete automated repair is indicated; the PR is ready for normal maintainer scope and merge review.

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Production and test delta	dashboard +686/-22, docs +82, tests +579/-8	The sizable dashboard implementation is paired with focused Worker, Durable Object, rendering, and privacy coverage.

Technical review

Best possible solution:

Land the current branch only after a maintainer confirms that the authenticated reconciliation surface and public telemetry-contract expansion match the intended operational scope.

Do we have a high-confidence way to reproduce the issue?

Not applicable: the PR addresses telemetry correctness through a supplied current-head local-container behavior trace rather than a separately reported runtime bug.

Is this the best way to solve the issue?

Yes. The branch keeps public surfaces observer-only while adding bounded authenticated reconciliation and explicit unavailable-state semantics.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 206fe726bbad.

Labels

Label changes:

• add rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• remove rating: 🐚 platinum hermit: Current PR rating is rating: 🦞 diamond lobster, so this older rating label is no longer current.

Label justifications:

• P2: This is a bounded dashboard telemetry correctness and observability improvement without evidence of an active user-facing outage.
• rating: 🦞 diamond lobster: Overall readiness is 🦞 diamond lobster; proof is 🦞 diamond lobster and patch quality is 🦞 diamond lobster.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (terminal): The supplied exact-head Crabbox local-container trace exercises the production Worker through rejected and signed reconciliation requests, public status/history contracts, and rendered Bay/Overview states with observed after-fix results.
• proof: sufficient: Contributor real behavior proof is sufficient. The supplied exact-head Crabbox local-container trace exercises the production Worker through rejected and signed reconciliation requests, public status/history contracts, and rendered Bay/Overview states with observed after-fix results.

Evidence

What I checked:

• Pinned PR range: The verified merge-base-to-head range changes 10 files by 1,347 additions and 30 deletions; current main is the exact base, so the telemetry-contract work is not already implemented there. (dashboard/worker.ts:1039, 311e64abb39a)
• Authenticated route boundary: The Worker authenticates the operator request, derives scope only from verified public repositories, and forwards it to the Durable Object reconciliation route. (dashboard/worker.ts:1039, 311e64abb39a)
• Reconciliation safeguards: The implementation bounds candidate scans and timing events, fails closed while lifecycle recovery is pending, and compares aggregate timing facts without returning lifecycle rows. (dashboard/exact-review-lifecycle-telemetry.ts:1079, 311e64abb39a)
• Behavior contract and proof: The committed contract specifies loopback Worker/Durable Object exercise, rejected and signed reconciliation calls, public status/history responses, and Bay/Overview rendering; the supplied PR trace records successful exact-head local-container results. (docs/proof/csw-143-telemetry-accuracy-foundation/behavior-contract.md:1, 311e64abb39a)
• Feature history: Available local history identifies Martin Cleary’s merged Bay review-journey timing work as the immediate feature foundation extended here. (dashboard/exact-review-lifecycle-telemetry.ts, 91a2bf645327)

Likely related people:

• Martin Cleary: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (4 earlier review cycles)

• reviewed 2026-08-29T23:58:16.612Z sha faa8af5 :: needs real behavior proof before merge. :: none
• reviewed 2026-08-30T01:40:53.406Z sha ca58ac2 :: needs maintainer review before merge. :: none
• reviewed 2026-08-31T10:32:28.324Z sha ea33fe7 :: needs real behavior proof before merge. :: [P2] Fail closed reconciliation during pending recovery | [P2] Provide the required local-container behavior proof
• reviewed 2026-08-31T11:56:32.161Z sha b276072 :: needs maintainer review before merge. :: none

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/33286296115
• Updated: 2026-08-30T01:43:05.409Z

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/33382998802
• Updated: 2026-08-31T10:34:31.959Z

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/33389459949
• Updated: 2026-08-31T11:58:48.490Z

[brokemac79]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/33390536234
• Updated: 2026-08-31T12:12:04.486Z