Skip to content

(MOT-4368) feat(e2e): surface semantic validation health - #729

Draft
ytallo wants to merge 1 commit into
mainfrom
feat/harness-validation-observability
Draft

(MOT-4368) feat(e2e): surface semantic validation health#729
ytallo wants to merge 1 commit into
mainfrom
feat/harness-validation-observability

Conversation

@ytallo

@ytallo ytallo commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Summary

  • fail fast on invalid Harness E2E manifests, lockfiles, providers, and expected binaries before cache, install, lint, test, or compilation work
  • publish schema 3 execution history with semantic statuses, exact run-attempt identity, and the first failed GitHub job or step when reports are unavailable
  • move latest operational health above efficiency and trends in the Pages dashboard
  • retain schema 2 compatibility while projecting all public detail through an allowlist that excludes prompts, transcripts, model responses, and tool payloads
  • paginate workflow-job diagnostics beyond 100 results

Why

Build and setup failures currently consume most of the workflow before exposing a lockfile or manifest problem, while the public dashboard can collapse distinct operational outcomes into a generic failure. Operators need a quick, safe view of what happened without publishing sensitive execution content.

Impact

The daily public lane gains actionable health and compact diagnostics. Quality-only misses remain advisory, hard and technical failures stay blocking, and rerun artifacts cannot be attached to the wrong attempt.

Validation

  • real preflight against 13 current source manifests
  • 5 manifest-preflight unit tests
  • 13 publisher cases and 8 collector semantic cases
  • 4 dashboard data/structure tests and 4 local server tests
  • Python compilation, workflow YAML parsing, actionlint, and diff checks

Refs MOT-4368

@vercel

vercel Bot commented Aug 6, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
workers Ready Ready Preview Aug 6, 2026 3:47pm
workers-tech-spec Ready Ready Preview Aug 6, 2026 3:47pm

Request Review

@coderabbitai

coderabbitai Bot commented Aug 6, 2026

Copy link
Copy Markdown

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 39f802d6-4710-47df-9100-4113ba7b7fe1

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant