test(uipath-insights): add e2e coverage for the jobs commands - #2551
test(uipath-insights): add e2e coverage for the jobs commands#2551celestinryf wants to merge 1 commit into
Conversation
The repo requires at least one e2e task per skill and uipath-insights had none. The five existing smoke tasks assert that command strings were attempted, regardless of outcome, so nothing graded whether uip insights jobs returns its shipped contract. Adds two e2e tasks with sidecar checkers plus a shared envelope helper. Scoped to the seven shipped Jobs commands, read-only. Teaches no proposed Filters, RBAC, or Alert commands as available.
|
Claude finished @celestinryf's task in 3m 14s —— View job PR Review: test(uipath-insights) — e2e coverage for jobs commands
SummaryAdds the first e2e coverage for Change-by-Change Review1.
|
| Area | Status | Notes |
|---|---|---|
| Frontmatter | N/A | No skill frontmatter changed |
| E2E Tests | OK | Two well-structured e2e tasks with valid tags, proper criteria weights, and sidecar checkers |
| Skill Body | N/A | SKILL.md and references untouched |
| References & Assets | N/A | No changes |
| Repo Hygiene | OK | No secrets, no @uipath/cli in env_packages, CODEOWNERS already covers the path, sys.path.insert follows repo convention |
Issues for Manual Review
- Tenant entitlement: Both tasks require a tenant that answers
uip insights jobs. The PR notes this and the checkers handle it with clear diagnostics, but whether the nightly runner's tenant is entitled for Insights queries is a runtime concern outside code review. - CLI Code strings: The
CODESdict inenvelope_check.pyis sourced fromcli/packages/insights-tool/src/commands/jobs.ts. If those Code strings change in a future CLI release, the checkers will fail. The author is aware (the docstring cites the source), but there's no automated sync — worth noting for maintenance.
Conclusion
Clean PR. The test design is thoughtful — data-agnostic validation, live probe for early tenant diagnostics, report grounding to prevent invented summaries, and proper timeout hierarchy. Code follows repo conventions throughout. No issues found that would require changes. Approve.
|
Claude finished @celestinryf's task in 5m 0s —— View job Coder-eval task lint (advisory)2 task YAMLs changed; verdict counts: 0 Critical, 0 High, 0 Medium, 0 Low, 2 OK. Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge. Evidence of passing run
Per-task lint
|
| walk(item) | ||
|
|
||
| if isinstance(data, dict): | ||
| walk(data.get("ProcessName")) |
There was a problem hiding this comment.
issue (non-blocking): data.get("ProcessName") pins the runtime-payload casing and fails silently — under a camelCase Data this returns [] while has_signal still reports signal, so _check_report skips the grounding branch and the task still scores 1.000 with the report ungraded against CLI output. The repo's existing fix for this class is the case-insensitive _get_ci accessor in uipath-maestro-flow/_shared/flow_check.py; tests/scripts/test_runtime_payload_key_casing.py won't flag it here since it only guards finalStatus/elementExecutions.
| - type: command_executed | ||
| description: "Agent ran summary with the 24-hour window (or an absolute epoch-ms range)" | ||
| tool_name: "Bash" | ||
| command_pattern: 'uip\s+insights\s+jobs\s+summary\s.*--(time-range[=\s]+1440|started-after[=\s]+\d{13})' |
There was a problem hiding this comment.
suggestion (non-blocking): .* crosses &&/;/|, so a neighbouring command in the same Bash invocation satisfies the flag — summary --time-range 60 > summary.json && ... completed-timeline --time-range 1440 ... passes this 24-hour-window criterion, and a summary with no time flag at all passes too. [^&;|\n]* (the form #2552 uses, or the lookahead style in uipath-test/link_automation_and_run.yaml) closes both; the 10 sibling patterns across the two YAMLs have the same shape.
4 participants
Related to IN-13526.
What
Adds the first e2e coverage for
uipath-insights: two task YAMLs with their sidecar checkers (envelope-contract/all_commands_envelope_e2e.yaml,job-health/job_health_investigation_e2e.yaml,check_all_commands_envelope.py,check_job_health_investigation.py) plus a shared_shared/envelope_check.pyholding the per-subcommand envelopeCodestrings fromjobs.ts. Five new files, all undertests/tasks/uipath-insights/. The five existing smoke tasks,SKILL.md, and the references are untouched.Why
The repo requires at least one smoke and one e2e task per skill (
tests/README.md:269) and this skill had only smoke. The five smoke tasks grade that command strings were attempted, regardless of outcome, so nothing currently grades whetheruip insights jobsreturns its shipped contract:Result: Successwith the exact per-subcommandCode.Scope
The seven shipped Jobs commands only, read-only, no tenant mutation and therefore no cleanup. Nothing here teaches the proposed Filters, RBAC, or Alert commands as available. The Filters skill content and its smoke task are a separate PR that waits on those commands reaching
cli/main.Testing
Ran both tasks locally and they passed. Run
2026-08-06_13-34-09, claude-code agent,uip 1.198.0, coder_eval 0.8.10:skill-insights-all-commands-envelope-e2e1.000 (12/12 criteria),skill-insights-job-health-investigation-e2e1.000 (9/9). Four earlier runs also passed at 1.000:2026-08-06_12-25-44,2026-08-06_12-13-08,2026-08-04_12-24-50,2026-08-04_12-29-42. The last run had live job data, so the report-grounding assertion was exercised rather than skipped through the empty-window path. Checker failure paths exercised locally against synthesized artifacts: ungrounded report, rejected envelope, unparseable envelope, missing file, nouipon PATH, probe timeout.scripts/check-task-driver.pyandscripts/check-cli-verbs.pyare clean.Limitations
No pytest unit tests for the checkers. maestro-flow, maestro-case and uipath-agents ship them, but running them in CI needs a
paths:entry and a job in.github/workflows/test-helpers.yml, outside this PR's ownership. Happy to follow up if the CI owners want it.The checkers verify that each saved envelope is structurally correct and that the report is grounded in the CLI's response, but not that a given file is the response to its own command. Both tasks need a tenant that answers
uip insights jobs; a tenant that rejects the read fails with the CLI's ownMessageandInstructionsrather than degrading.How to Review
Start with the two
check_*.pyfiles and_shared/envelope_check.py. That is where the contract lives. The YAMLs are mostly criteria lists.Checklist
CI note
The PR gate runs only
smoke-tagged tasks, so neither new task is exercised by CI on this PR. The local runs above are the evidence; nightly is where they run on a schedule.