test(flow/hitl): path-agnostic criteria for solution-wrapped flow projects

[nikhil-maryala]

Problem

Agents that scaffold via uip solution init produce <Name>Solution/<Name>/<Name>.flow, but 17 greenfield task YAMLs hardcode the flat <Name>/<Name>/<Name>.flow in file_exists / file_contains / file_matches_regex / run_command criteria. A correct, validating flow scores 0.0 purely on the path. Nightly evidence: the 7/30 delegate run's own analysis.md documents 14 HITL/flow tasks failing exactly this way; the 7/31 claude run flagged 4 more (hitl-quality-result-downstream, hitl-quality-priority-timeout, paginated-reference-lookup, ixp-scaffold-multinode); antigravity runs show the same signature as validate: File not found. Because the wrapper choice is stochastic per model, this mismatch manufactures fake cross-model signal.

The already-migrated hitl/smoke_01..03 tasks established the fix pattern (#2213's validate_flow.py + flow_contains.py discovery); this PR applies it to the remaining floating tasks.

Changes

• _shared/flow_contains.py: add repeatable --regex PATTERN — the path-agnostic replacement for file_matches_regex. New test_flow_contains.py covers substring/regex present/absent, the negative-check exit-1 contract, and dangling-flag exit 2 (75/75 pass locally).
• 61 criteria across 17 task YAMLs migrated to validate_flow.py / flow_contains.py discovery. Descriptions, weights, and pass_thresholds preserved verbatim.
• quality_08_variable_binding_fieldid's negative check (script must NOT reference .output.legalApproval|legalNotes) keeps its inverted contract: expected_exit_code: 1 with a comment explaining the semantics.

Intentionally untouched: tasks whose prompts pin the literal path (e2e_02/04/05/07 brownfield fixtures, inline_agent_robust) — there the hardcoded path is the contract, not a bug.

Verification

• Unit tests: 75/75 in tests/tasks/uipath-maestro-flow/_shared/ (CI: test-helpers.yml).
• run-coder-eval.yml dispatched from this branch on the evidence-backed regressing tasks across claude / codex / antigravity — run links + results to follow in a comment. Note the wrapper choice is stochastic: green runs prove no regression on both layouts; the unit tests prove the discovery works for the wrapped layout deterministically.

🤖 Generated with Claude Code

[github-actions]

Claude finished @nikhil-maryala's task in 1m 26s —— View job

---

Coder-eval task lint (advisory)

• Read lint rubric (.claude/commands/lint-task.md)
• Identify changed task YAMLs (17 files)
• Check PR body for evidence of passing run
• Apply rubric to each changed file
• Within-PR duplicate check
• Post final lint comment

[github-actions]

Claude finished @nikhil-maryala's task in 2m 9s —— View job

---

PR Review: test(flow/hitl): path-agnostic criteria for solution-wrapped flow projects

• Read review criteria and project conventions
• Read full diff
• Read existing PR conversation
• Review flow_contains.py changes
• Review test_flow_contains.py (new file)
• Review HITL task YAML migrations
• Review maestro-flow task YAML migrations
• Check for cross-skill references, secrets, broken links
• Post final review

---

Summary

Migrates 61 success criteria across 17 task YAMLs (10 HITL, 7 maestro-flow) from hardcoded file_exists/file_contains/file_matches_regex checks to the path-agnostic flow_contains.py/validate_flow.py discovery helpers. Also extends flow_contains.py with --regex PATTERN support and adds a 7-case unit test file. This fixes false-negative scoring caused by stochastic <Name>Solution/ wrapper directory scaffolding.

Change-by-Change Review

1. tests/tasks/uipath-maestro-flow/_shared/flow_contains.py

Severity: OK

Clean addition of --regex flag parsing via iter(argv). The arg parser correctly handles:

• Mixed substring and regex args in any order
• Dangling --regex flag → exit 2
• Exit 1 when any substring or regex is absent

The re.search() call uses uncompiled patterns, which is fine for single-use. File handles opened in the bodies dict comprehension are not explicitly closed, but this matches the pre-existing pattern and is acceptable for short-lived CLI scripts.

Minor nit: the FAIL message always prints both counts (0 substring(s) and 1 regex(es)), which is slightly noisy when only one type fails. Not worth changing.

2. tests/tasks/uipath-maestro-flow/_shared/test_flow_contains.py (new)

Severity: OK

Good coverage of the new functionality: 7 tests covering existence-only, substring present/absent, regex present/absent, mixed mode, and dangling flag. The sandbox fixture uses the solution-wrapped layout (DemoSolution/Demo/) which directly exercises the discovery path this PR targets. Clean pytest idioms.

3. HITL task YAMLs (10 files)

Severity: OK

All 10 HITL task files follow the same mechanical pattern:

• file_exists → run_command calling flow_contains.py with no args
• file_contains → run_command calling flow_contains.py with substring args
• file_matches_regex → run_command calling flow_contains.py --regex
• Hardcoded uip maestro flow validate <path> → validate_flow.py
• Descriptions, weights, and pass_threshold preserved verbatim

The quality_08_variable_binding_fieldid.yaml negative check (expected_exit_code: 1) is correctly preserved with a clear comment explaining the semantics.

4. Maestro-flow task YAMLs (7 files)

Severity: OK

Same mechanical migration. paginated_reference_lookup.yaml, devcon_expense_approval.yaml, 4 HITL quality tasks, and scaffold_multinode.yaml all follow the established pattern.

5. scaffold_multinode.yaml — removed run_limits

Severity: Medium

The run_limits block was removed:

run_limits:
  expected_turns: 24
  max_turns: 40
  turn_timeout: 900
  task_timeout: 900

This causes the task to fall back to experiment defaults (max_turns: 200, task_timeout: 1200), which is significantly more permissive — 5x the max turns and 33% more timeout. If the prior limits were intentional constraints for this task, this is a behavioral change. The PR description does not mention this removal.

Fix this →

6. Cross-skill helper usage

Severity: OK

The HITL tasks reference $SKILLS_REPO_PATH/tests/tasks/uipath-maestro-flow/_shared/ helpers. This is a test-infrastructure cross-reference (shared test helpers), not a structural cross-skill dependency in the skill definitions themselves. The _shared/ directory is purpose-built for this reuse pattern.

What's Missing

• scaffold_multinode.yaml run_limits removal is undocumented. The run_limits block was silently dropped. If intentional (inheriting from experiment defaults), worth noting in the PR description. If accidental, should be restored.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	No skill frontmatter changes
E2E Tests	OK	Criteria migrated correctly, weights/thresholds preserved
Skill Body	N/A	No skill doc changes
References & Assets	OK	Shared helper extended cleanly with tests
Repo Hygiene	OK	No secrets, no broken links, scoped changes

Issues for Manual Review

• Verify whether the run_limits removal from scaffold_multinode.yaml was intentional. The prior limits (max_turns: 40) were significantly tighter than the experiment default (max_turns: 200).
• The PR description mentions dispatched coder-eval runs but results are pending — confirm those pass before merging.

Conclusion

Clean, well-scoped mechanical migration that follows the established pattern from #2213. The flow_contains.py extension is minimal and well-tested. One Medium finding: the run_limits removal from scaffold_multinode.yaml appears accidental and should be either restored or justified.

---

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7a5c251a24

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Scope discovery to the requested flow name

When an agent creates any lone Flow project, this criterion succeeds regardless of its name because flow_contains.py discovers an arbitrary Flow project without receiving the expected InvoiceApproval basename. A correctly shaped Foo.flow can therefore earn all artifact and validation points while ignoring the prompt's naming requirement that the previous literal path enforced; keep the wrapper directory flexible but require the expected project/flow basename.

Useful? React with 👍 / 👎.

[nikhil-maryala]

Fixed in 0b9d18a — every migrated invocation now passes --flow-name <Name> (names recovered from the original literal paths), so a lone Foo.flow no longer earns the points: discovery stays wrapper-agnostic but the basename must match. Covered by test_flow_name_mismatch_exits_1 / test_flow_name_scopes_assertions.

[chatgpt-codex-connector]

Match each assertion set within one flow file

When a project contains multiple flows or subflows, each requested substring or regex is searched independently across all files, so no single flow needs to satisfy the criterion. For example, the migrated scaffold_multinode.yaml check for postReceipt, sendToValidation, and logError passes if those names are split among three valid flows even though the task requires all three nodes in the authored ReceiptIntake flow; require one discovered body to satisfy the complete assertion set.

Useful? React with 👍 / 👎.

[nikhil-maryala]

Fixed in 0b9d18a — positive assertion sets (substrings + --regex) must now be satisfied by ONE discovered file; split-matching across subflows fails with FAIL: no single .flow file satisfies the full assertion set. Combined with --flow-name ReceiptIntake on the scaffold_multinode criteria, the exact scenario you describe now scores 0. Covered by test_assertion_set_must_match_single_file.

[chatgpt-codex-connector]

Distinguish successful absence from helper failure

When the agent creates no .flow file, discovery is ambiguous, or reading a flow fails, flow_contains.py exits 1—the same code this negative criterion treats as success. Because criteria are scored independently, the other existence checks do not prevent an invalid submission from receiving these 2.5 points; use an explicit absence assertion that returns success only after project discovery and file reads complete without the forbidden pattern.

Useful? React with 👍 / 👎.

[nikhil-maryala]

Fixed in 0b9d18a — added an explicit --absent-regex mode that exits 0 only when discovery succeeded, the --flow-name ContractReview file was read, and the forbidden pattern is absent. quality_08's negative check now uses it with expected_exit_code: 0, so a missing/ambiguous flow can never collect the 2.5 points. Covered by test_absent_regex_fails_without_flow.

[nikhil-maryala]

Passing-run claim — run-coder-eval.yml from this branch, 6 migrated tasks (flow/hitl quality_01–04, hitl quality_07/08) per agent:

Agent	Model	Result	Run
claude	claude-sonnet-5	✅ 6/6 at 1.0	31349867382
codex	gpt-5.6-terra	✅ 6/6 at 1.0	31349868685
antigravity	gemini-3.5-flash	1/6 — pre-existing model baseline, see below	31349869793

Antigravity attribution (all 5 failures are the flash-tier authoring collapse these tasks already show in the nightlies — none are caused by this migration):

• boolean-decision is the fix working as intended: discovery reports Found 1 .flow file(s): VendorApprovalSolution/VendorApproval/VendorApproval.flow — the solution-wrapped layout that previously scored 0.0 on path alone — and fails only on missing content (no boolean output field authored).
• schema-design / runtime-vars: No project.uiproj found — the agent produced no project at all; the old hardcoded-path criteria would have failed identically.
• result-downstream / variable-binding-fieldid: flow found (flat layout), required HITL content absent.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0b9d18a3a8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[chatgpt-codex-connector]

Restore the task's custom run limits

Restore the run_limits block after this migrated criterion. Without it, ixp-scaffold-multinode inherits the default experiment's max_turns: 200 and task_timeout: 1200 instead of its previous 40-turn/900-second limits, allowing substantially longer and more expensive runs and changing pass-rate comparability for a commit intended only to make artifact paths flexible.

Useful? React with 👍 / 👎.

[nikhil-maryala]

Re-verification after the review-round commit (0b9d18a3a — --flow-name scoping, single-file matching, --absent-regex):

Agent	Model	Result	Run
claude	claude-sonnet-5	✅ 6/6 at 1.0	31444647425
codex	gpt-5.6-terra	✅ 6/6 at 1.0	31444649386

Tasks: flow/hitl quality_01–04 + hitl quality_07/08 — covers name-scoped positive assertions and the migrated --absent-regex negative check in quality_08. Unit tests: 83/83 (CI maestro-flow checker unit tests green).