test(hitl): stop gating validate checks on the literal --output json flag

[nikhil-maryala]

Problem

Ten command_executed criteria in the uipath-human-in-the-loop suite gate "Agent validated the flow" on the pattern (uip|\$UIP)\s+(maestro\s+)?flow\s+validate.*--output\s+json. The --output json flag is outcome-invisible, so the pattern docks agents that reach the same validated result without typing the flag — .claude/rules/test-writing.md forbids exactly this gate ("Never add a gating command_executed check on --output json"). Observed in the wild: the #2534 verification run (antigravity) scored Matched 0/1 on this criterion.

Related cases already fixed on main: flow-solution-select-ask no longer gates on the Claude-Code-only AskUserQuestion tool, and the evaluate/* flow tasks dropped their --output json gates — this PR closes out the last evidence-backed instances in the flow/hitl suites.

Fix

Drop .*--output\s+json from the gating pattern in all 10 YAMLs; the criterion still gates on the behavior (flow validate executed). One line per file, no weights or thresholds changed.

Out of scope (for owners)

uipath-test (10 criteria) and uipath-coded-apps (4 criteria) embed --output json in their gating command patterns too. Left untouched — different CODEOWNERS; flagging for those owners to decide.

Verification

run-coder-eval.yml from this branch on quality_04_all_handles, quality_07_runtime_vars, e2e_06_invoice_approval_greenfield_simple across claude / codex / antigravity — links to follow in a comment.

🤖 Generated with Claude Code

[github-actions]

Claude finished @nikhil-maryala's task in 1m 13s —— View job

---

PR Review: test(hitl): stop gating validate checks on the literal --output json flag

• Read review criteria and project conventions
• Read full diff
• Read existing PR conversation
• Read full files for context
• Post review

---

Summary

Drops .*--output\s+json from the command_pattern in 10 command_executed criteria across the uipath-human-in-the-loop test suite. The pattern now gates only on the behavior (flow validate executed), not on the outcome-invisible --output json flag — exactly what .claude/rules/test-writing.md requires.

Change-by-Change Review

1. All 10 YAML files — command_pattern simplification

Severity: OK

Every change is identical and mechanical: '(uip|\$UIP)\s+(maestro\s+)?flow\s+validate.*--output\s+json' → '(uip|\$UIP)\s+(maestro\s+)?flow\s+validate'. No weights, thresholds, descriptions, or other fields changed. The run_command criteria in these same files (which do use --output json in their command: field) are untouched — correct, since those are harness-side commands, not agent-behavior gates.

Files changed:

• e2e_01_invoice_approval_greenfield.yaml:72
• e2e_02_ai_escalation_brownfield.yaml:62
• e2e_03_gdpr_compliance_greenfield.yaml:64
• e2e_04_multi_hitl_brownfield.yaml:67
• e2e_05_expense_approval_brownfield.yaml:75
• e2e_06_invoice_approval_greenfield_simple.yaml:63
• e2e_07_apptask_brownfield.yaml:67
• quality_04_all_handles.yaml:46
• quality_05_priority_and_timeout.yaml:41
• quality_07_runtime_vars.yaml:37

Verified: after this PR, grep command_pattern.*--output across tests/tasks/uipath-human-in-the-loop/ returns zero results. All remaining --output json references in the directory are inside run_command criteria (harness-executed), which is correct.

What's Missing

Nothing identified. The PR description transparently scopes out the uipath-test (10 criteria) and uipath-coded-apps (4 criteria) instances for separate ownership — reasonable given different CODEOWNERS.

Area Ratings

Area	Status	Notes
Frontmatter	OK	No frontmatter changes
E2E Tests	OK	Criteria patterns align with .claude/rules/test-writing.md; no weights/thresholds changed
Skill Body	OK	No skill files changed
References & Assets	OK	No references changed
Repo Hygiene	OK	Scoped to HITL suite only; no secrets, no unrelated changes

Issues for Manual Review

None found. The run_command criteria already independently verify that uip flow validate passes (with --output json in the harness command), so removing the flag from command_pattern does not reduce test coverage.

Conclusion

Clean, mechanical fix that aligns 10 test tasks with the repo's own test-writing rules. No issues found — approve.

[github-actions]

Claude finished @nikhil-maryala's task in 3m 1s —— View job

---

Coder-eval task lint (advisory)

10 task YAMLs modified; verdict counts: 0 Critical, 0 High, 0 Medium, 0 Low, 10 OK (1 pre-existing theme noted below).

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

✅ Author confirms verification: "run-coder-eval.yml from this branch on quality_04_all_handles, quality_07_runtime_vars, e2e_06_invoice_approval_greenfield_simple across claude / codex / antigravity — links to follow in a comment." Three representative tasks cover the identical one-line change applied to all 10 files.

Per-task lint

All 10 files received the same mechanical edit (removing .*--output\s+json from the command_executed command_pattern). Each file was evaluated against all seven rubric axes. No file-specific issues were found beyond the shared theme below.

tests/tasks/uipath-human-in-the-loop/e2e_01_invoice_approval_greenfield.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

tests/tasks/uipath-human-in-the-loop/e2e_02_ai_escalation_brownfield.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

tests/tasks/uipath-human-in-the-loop/e2e_03_gdpr_compliance_greenfield.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

tests/tasks/uipath-human-in-the-loop/e2e_04_multi_hitl_brownfield.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

tests/tasks/uipath-human-in-the-loop/e2e_05_expense_approval_brownfield.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

tests/tasks/uipath-human-in-the-loop/e2e_06_invoice_approval_greenfield_simple.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

tests/tasks/uipath-human-in-the-loop/e2e_07_apptask_brownfield.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

tests/tasks/uipath-human-in-the-loop/quality_04_all_handles.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

tests/tasks/uipath-human-in-the-loop/quality_05_priority_and_timeout.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

tests/tasks/uipath-human-in-the-loop/quality_07_runtime_vars.yaml — verdict: OK (pre-existing theme noted; see Theme 1)

Within-PR duplicates

No duplicate clusters detected. All 10 files are pre-existing tasks with distinct coverage (greenfield vs brownfield, single vs multi-node, schema design vs wiring vs runtime vars vs priority options vs AppTask surface).

Themes

• Theme 1 (pre-existing, not introduced by this PR): Validate-only flow tests miss correctness. All 7 e2e-tagged tasks run flow validate but not flow debug. All 3 integration-tagged quality tasks do the same. Every e2e task's description and inline YAML comment explicitly documents this: "Does not deploy or run the flow" / "e2e here means authoring end-to-end". The description-rationale carve-out applies (High → Medium for e2e, Medium stays for integration), and the theme-aware downgrade reduces per-task severity further (→ Low). Since this is a deliberate design choice across the entire HITL suite — not introduced or worsened by this PR — per-task verdicts are left at OK. Addressing this would be a separate initiative.

Conclusion

✅ All 10 changed tasks pass the rubric for the change under review. The edit correctly removes the --output json gate from command_executed patterns, aligning with .claude/rules/test-writing.md ("Never add a gating command_executed check on --output json"). Each task retains a run_command criterion that independently validates flow validate succeeds — no coverage is lost.

---

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 250dd56e3c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Require a target before counting validation

When an agent probes the CLI with uip maestro flow validate --help, this shortened regex now counts the command as “validated” even though no flow file was checked. The canonical invocation requires <file> (skills/uipath-human-in-the-loop/SKILL.md:39), and the later run_command is grader-owned rather than evidence that the agent performed validation, so all ten changed tasks can incorrectly award this behavioral criterion during command discovery; require a real positional target or explicitly exclude help invocations while leaving --output json optional.

Useful? React with 👍 / 👎.

[jiyangzh]

+1, is the change making it too loose.

[nikhil-maryala]

Passing-run claim — run-coder-eval.yml from this branch (quality_04_all_handles, quality_07_runtime_vars, e2e_06_invoice_approval_greenfield_simple):

Agent	Model	Result	Runs
claude	claude-sonnet-5	✅ 3/3 at 1.0	31444192092 (2/3; one 600s agent turn-timeout before any criteria ran) + retry 31445116051 (1.0)
codex	gpt-5.6-terra	✅ 3/3 at 1.0	31444193671
antigravity	gemini-3.5-flash	0/3 — pre-existing flash-tier baseline	31444195129

Antigravity note: the agent authored no HITL node and never ran validate — its "Agent validated the flow" criterion now correctly reports Matched 0/1 against the new flag-free pattern, i.e. the behavior gate still works; only the outcome-invisible --output json requirement is gone.