Align Maestro Flow tasks for same-ground evaluation

[tmatup]

Summary

• align the nine path-to-GA Maestro Flow tasks on one loop-neutral prompt, criterion, weight, limit, and task-id contract
• add five report-only structural advisories with zero pass thresholds
• add a campaign-only overlay for the pinned image, Sonnet 5 route, shared system prompt, environment surface, Proactive output style, isolated auth home, and an empty read-only catalog shadow
• make the pinned Docker image install the stable CLI from public npm before its private tool plugins, with the npm credential delivered only through a BuildKit secret

Nightly-signal rationale

The task files intentionally change the future nightly signal:

• the invoice-extraction task loses 5.5 criterion-weight points that were outside the approved intersection
• tasks 2–6 no longer forbid the agent from running debug itself, so the skill owns its verification loop
• run limits use the predeclared shared envelopes rather than the previous task-local limits

These are measurement-contract changes, not greening changes. The committed nightly experiment is untouched; all campaign-specific harness deltas live in the new overlay.

Validation

• same-ground sync gate: 9/9 pairs
• shared Flow checker tests: 68/68
• skill composition validation: 25 skills, 1,706 files, both flavors
• all nine changed tasks executed in the branch shakeout; every observed red is retained and classified, with no run-driven prompt, checker, weight, or limit change
• final pinned-image pagination canary: 5/5 criteria, score 1.000; Sonnet 5, Proactive, auth, plugin delivery, pagination, and catalog isolation green
• image inventory: Claude Code 2.1.177, coder-eval 0.9.5, UiPath CLI 1.198.0, 29 tool plugins; npm credential absent from image history
• prompt audit: no resource GUIDs or task-specific coaching
• latest-head Claude review complete; every severity and manual-review item triaged

🤖 Generated with Claude Code
Co-Authored-By: Claude

[github-actions]

Claude finished @tmatup's task in 4m 56s —— View job

---

Coder-eval task lint (advisory)

9 task YAMLs changed; 0 Critical, 0 High, 3 Medium, 0 Low, 6 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

⚠️ High — PR body claims one specific passing run ("first local pinned-image pagination canary: 5/5 criteria, score 1.000") but does not assert that the remaining 8 changed tasks have been run and passed. The "same-ground sync gate: 9/9 pairs" is a structural parity check, not a task-run claim. Please consider editing the PR description to add a line like:

Ran all 9 changed tasks locally and they passed.

or per-task claims for the ones that were exercised.

Per-task lint

tests/tasks/uipath-maestro-flow/connector_features/drive_to_slack.yaml — verdict: Medium (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-flow/connector_features/paginated_reference_lookup.yaml — verdict: Medium (theme-captured; see Theme 1)

tests/tasks/uipath-maestro-flow/ixp/e2e_01_invoice_extraction_greenfield.yaml — verdict: Medium

Issues:

• [Medium] Validate-only flow tests miss correctness: tagged e2e (line 10) with no flow debug execution. Description-rationale carve-out applied (lines 7–8: "Validate-only: no uip maestro flow debug. IxP runtime + SharePoint connector both require a tenant deployment which CI does not have") — severity downgraded from High to Medium per rubric.

Suggested fixes:

• No fix needed if the description rationale accurately reflects the constraint. Consider whether a future CI environment with tenant access could upgrade this to a debug-verified test.

tests/tasks/uipath-maestro-flow/multi_node/billing_discrepancy_detector/billing_discrepancy_detector.yaml — verdict: OK

tests/tasks/uipath-maestro-flow/multi_node/billing_dispute_analyst/billing_dispute_analyst.yaml — verdict: OK

tests/tasks/uipath-maestro-flow/multi_node/billing_dispute_resolution/billing_dispute_resolution.yaml — verdict: OK

tests/tasks/uipath-maestro-flow/multi_node/billing_invoice_lookup/billing_invoice_lookup.yaml — verdict: OK

tests/tasks/uipath-maestro-flow/multi_node/billing_resolution_writer/billing_resolution_writer.yaml — verdict: OK

tests/tasks/uipath-maestro-flow/single_node/coded_agent/coded_agent.yaml — verdict: OK

Within-PR duplicates

No duplicate clusters detected. The five billing_* tasks decompose distinct sub-operations of the DevCon BillingDisputeResolution scenario (parallel DS lookups, inline agent + SOP index grounding, full pipeline, invoice normalization + query, agent email drafting) — materially distinct operations sharing a scaffold, not interchangeable tests.

Themes

• Theme 1 [High] — Validate-only e2e flow tests without description rationale (2 tasks): drive_to_slack.yaml (line 13: tag e2e) and paginated_reference_lookup.yaml (line 10: tag e2e) are tagged e2e but neither the agent nor the check scripts invoke flow debug, and the description field does not document why debug is skipped. Per-task verdicts downgraded to Medium. Suggested fix: add a rationale to each task's description field (e.g. "Validate-only: no flow debug — requires live Google Drive / Slack credentials not available in CI.") to trigger the description-rationale carve-out, or reconsider whether integration alone is the appropriate tier tag if the test's value is in authoring-time behavior rather than runtime execution.

Conclusion

⚠️ 3 task(s) have issues, max severity High (in theme). Evidence of passing run is partial (1 of 9 tasks). Advisory only — not blocking merge.

[github-actions]

Claude finished @tmatup's task in 4m 8s —— View job

---

PR Review: Align Maestro Flow tasks for same-ground evaluation

• Read review criteria and project conventions
• Analyze full diff (18 changed files)
• Review Dockerfile changes
• Review new experiment overlay
• Review new advisory checker scripts
• Review task YAML changes (9 task files)
• Check for security issues, secrets, hardcoded paths
• Check cross-skill references and self-containment
• Post final review

---

Summary

Aligns nine Maestro Flow task YAMLs to a shared same-ground evaluation contract: unified prompt conventions, criterion weights at the v1-base intersection, report-only structural advisories at pass_threshold: 0.0, and a campaign-only experiment overlay. Also adjusts the Docker image to install the CLI shell from public npm before writing the private GitHub Packages npmrc.

Change-by-Change Review

1. tests/docker/Dockerfile

Severity: OK

Reorders npm install -g @uipath/cli to run before the @uipath→GitHub Packages npmrc is written, so stable CLI releases that aren't mirrored to GHP can install from the public registry. The tool plugins continue from the private feed. Also adds a LABEL org.uipath.cli.version for image introspection. The npmrc is still created and removed within the same RUN layer — no credential leak. Logic is sound.

2. tests/experiments/same-ground-headtohead.yaml (new)

Severity: Medium — hardcoded personal path

The experiment overlay is well-structured: pinned Docker image, Sonnet 5 route, outputStyle: Proactive, shared system prompt, scoped allowed tools.

However, line 40 hardcodes a personal home directory:

extra_mounts:
  - ~/.uipath:/home/tmatup/.uipath:rw

The existing nightly.yaml and smoke.yaml both use ~/.uipath:/.uipath:rw (portable). The /home/tmatup target path is a personal filesystem path that will break for any other developer or CI runner. Fix this →

3. Five new advisory checker scripts (_shared/advisory_billing_*.py)

Severity: OK

Five well-documented structural gate scripts for the billing task family. Each validates flow topology, wiring, contract declarations, and anti-hardcode invariants that behavioral debug runs cannot see. Common patterns:

• fail() exits with a diagnostic FAIL: message
• unwrap() handles expression envelopes uniformly
• Each check section is numbered with clear doc comments

Code quality is high — the scripts are thorough, the error messages are actionable (they explain why the assertion matters), and they don't share imports across _shared/ files (self-contained). The unwrap() function is duplicated across all five scripts, which is a minor DRY concern but acceptable given the self-containment constraint on checker scripts.

4. _shared/check_inline_agent.py

Severity: OK

Default glob pattern changes from */*/*/agent.json to **/agent.json with recursive=True. This makes the checker robust to varying directory depths — a good fix.

5. connector_features/check_drive_to_slack.py

Severity: OK

Removes the unused run_debug import from _shared.flow_check and the os import + sys.path manipulation. Clean — the debug call was removed from the corresponding task's criteria (now structural-only), so the import was dead code.

6. connector_features/drive_to_slack.yaml

Severity: Low — style nit on initial_prompt

Criteria preserved with the same weights. The criterion description drops "and debug step finished green" — consistent with the structural-only scope. The initial_prompt was changed from a YAML block scalar (|) to a double-quoted string with embedded \n, which is harder to read and edit. All other task YAMLs in this PR correctly use | for multi-line prompts. Functionally equivalent but inconsistent with the rest of the PR.

Tags switched from inline [...] to block list style — fine, consistent with the rest.

Added the "Do NOT ask for approval" suffix to the prompt — aligns with the loop-neutral contract.

7. connector_features/paginated_reference_lookup.yaml

Severity: OK

Whitespace normalization (inline tags → block list, indentation). Criteria preserved with identical weights, patterns, and thresholds. No functional change.

8. ixp/e2e_01_invoice_extraction_greenfield.yaml

Severity: OK

Drops 3 criteria (total 8 → 5): skill_triggered (weight 2.0), command_executed for IxP registry search (weight 2.0), and command_executed for SharePoint registry search (weight 1.5). This is the 5.5-point reduction described in the PR body. The remaining criteria (flow validate, IxP node presence, SharePoint presence, HTTP node, $vars reference) form the structural intersection. Adds the "Do NOT ask" suffix to the prompt. Removes inline run_limits comments.

9. Five multi_node/billing_* task YAMLs

Severity: OK

All five follow the same pattern:

• Retain the v1-base criteria (validate + domain checker) with original weights
• Add 1–2 report-only advisory criteria (pass_threshold: 0.0, weight: 1.0) pointing to the new _shared/advisory_billing_*.py scripts
• Two tasks (billing_discrepancy_detector, billing_invoice_lookup) also add a bindings.json stub-check advisory
• Remove "Do not run debug yourself" from prompts (now "loop-neutral" — agent can debug if useful)
• billing_dispute_analyst and billing_resolution_writer raise task_timeout from 2400 → 3300

Two tasks (billing_discrepancy_detector at lines 51-62, billing_invoice_lookup at lines 47-58) embed identical inline Python for the bindings.json stub-UUID check. This is duplicated verbatim — could be a shared script in _shared/, but since these are inline advisory checks with pass_threshold: 0.0, the duplication is minor.

10. single_node/coded_agent/coded_agent.yaml

Severity: OK

Adds max_turns: 60 and task_timeout: 2400 (previously inherited from experiment defaults). Whitespace normalization only on criteria and post_run.

What's Missing

• The coded_agent.yaml task is missing mode:build in its tags — but this predates this PR (not a regression).
• No mode:build tag on coded_agent is a known pre-existing gap, not introduced here.
• Nothing else identified.

Area Ratings

Area	Status	Notes
Frontmatter	OK	All task YAMLs have valid structure
E2E Tests	OK	No new skills — test coverage not applicable
Skill Body	N/A	No skill SKILL.md files changed
References & Assets	OK	New advisory scripts are well-structured
Repo Hygiene	Issue	Personal path /home/tmatup in experiment overlay (line 40)

Issues for Manual Review

• Bindings.json advisory inline code — the identical Python snippet embedded in billing_discrepancy_detector.yaml and billing_invoice_lookup.yaml should be verified to produce the correct exit code when bindings.json doesn't exist (it will raise FileNotFoundError, which is exit code 1 — correct behavior for a failing advisory, but the error message won't be the clean FAIL: format the other advisories use).
• eu.anthropic.claude-sonnet-5 model identifier — verify this is a valid Bedrock model ID for the campaign's routing. Not verifiable from the repo alone.
• skills-image:sg1 docker image tag — this is a locally-built image name; confirm the campaign's build pipeline tags images this way.

Conclusion

Clean, well-documented alignment of nine task files to a shared evaluation contract. The changes are consistent across tasks and the new advisory scripts are thorough. One actionable finding: the hardcoded personal path /home/tmatup/.uipath in same-ground-headtohead.yaml:40 should use the portable /.uipath target that nightly.yaml and smoke.yaml use. The drive_to_slack.yaml prompt style (quoted string vs block scalar) is a minor inconsistency. Otherwise, approve.
|

[tmatup]

Review triage on the current head:

• Medium — campaign overlay mount: no one-arm edit. coder-eval forwards the host HOME into Docker, while extra_mounts expands variables only on the source side. On this pinned runner, changing the destination to /.uipath would make the CLI miss its auth cache. The overlay is runner-specific by design; the underlying portability gap is filed as coder_eval #100.
• Medium — validate-only e2e tasks: expected under the approved intersection contract. Adding runtime/debug scoring criteria would change the measurement contract; no checker or criterion change.
• Low — quoted prompt scalar: the decoded prompt bytes are the contract and are intentionally unchanged. The YAML representation does not alter the agent input; no functional edit.
• Passing-run evidence: the PR body deliberately claims only the completed pagination canary. It will be updated only after the full nine-task shakeout is actually complete.
• Docker default dev train: the first CI run exposed a separate registry-branch regression. Current head 3d69f446f keeps stable numeric releases on public npm and routes prerelease pins/dist-tags through the authenticated package feed.

No severity was skipped; the only code-level CI finding is fixed on the current head.

[tmatup]

@claude review

[github-actions]

PR Review: Align Maestro Flow tasks for same-ground evaluation

Reviewing PR changes...

• Read review criteria and project conventions
• Analyze full diff
• Review Dockerfile changes
• Review workflow changes
• Review experiment overlay
• Review task YAML changes
• Review advisory checker scripts
• Review shared checker changes
• Check for security issues, secrets, cross-skill dependencies
• Post final review

[tmatup]

@bai-uipath @rockymadden The path-to-GA approval check requires an allowlisted approval on the current head. Could one of you please review and approve when available?

[tmatup]

Current head 3979fc089 adds the symmetric read-only catalog shadow after a branch canary proved that uip skills list could repopulate the auth cache. Direct probes on both pinned arm images retain authenticated uip access, make the extra catalog fail closed, and leave the shadow empty.

@claude review

🤖 Generated with Claude Code
Co-Authored-By: Claude

[tmatup]

@bai-uipath @rockymadden The allowlisted path-to-GA approval is still the only policy gate. The latest head is 3979fc089; please review and approve this head when available.

🤖 Generated with Claude Code
Co-Authored-By: Claude

[github-actions]

Claude finished @tmatup's task in 3m 50s —— View job

---

PR Review: Align Maestro Flow tasks for same-ground evaluation

• Read review criteria and project conventions
• Get full diff and read changed files
• Review Dockerfile changes
• Review experiment overlay
• Review task YAML changes (9 tasks + 2 connector_features)
• Review advisory checker scripts (5 new + 2 modified)
• Review CI workflow changes
• Review skill doc changes (4 files)
• Check for secrets, cross-skill refs, and broken links
• Post findings

---

Summary

Aligns nine Maestro Flow task YAMLs to a shared same-ground evaluation contract (loop-neutral prompts, v1-base criterion weights, report-only structural advisories at pass_threshold: 0.0), adds a campaign-only experiment overlay, migrates the Docker build to BuildKit secrets for the npm credential, adds five structural advisory checker scripts, and corrects array-field authoring guidance in the connector skill docs. The latest commit (3979fc089) adds a read-only empty .skills shadow mount to block auth-cache repopulation via uip skills list.

Change-by-Change Review

1. .github/workflows/run-coder-eval.yml + .github/workflows/smoke-skills.yml

Severity: OK

Both workflows switch from --build-arg NPM_AUTH_TOKEN=… to --secret id=npm_auth_token,env=NPM_AUTH_TOKEN. The token is now exposed via an env: block and consumed by a BuildKit secret mount — it no longer appears in docker history or build output. Correct and a meaningful security improvement.

2. tests/docker/Dockerfile

Severity: OK

Well-documented rewrite. Stable numeric CLI releases install from public npm before the @uipath→GHP npmrc is written, then prerelease/dev pins install from GHP. The npmrc is written and removed within a single RUN layer, and the credential arrives via --mount=type=secret,id=npm_auth_token,required=true — no ARG exposure. The MIN_TOOLS floor, dist-tag derivation logic, and LABEL addition are all sound.

3. tests/experiments/same-ground-headtohead.yaml

Severity: Medium — personal path (previously acknowledged)

Lines 42–43:

extra_mounts:
  - ${SG_UIPATH_HOME}:/home/tmatup/.uipath:rw
  - ${SG_EMPTY_SKILLS}:/home/tmatup/.uipath/.skills:ro

The /home/tmatup target path is a personal filesystem path. The author previously acknowledged this is runner-specific by design, with the underlying portability gap tracked as coder_eval#100. The SG_EMPTY_SKILLS read-only shadow mount (latest commit) is a correct approach to blocking uip skills list from writing to the auth cache — Docker gives the more-specific child mount path precedence over the parent rw mount.

No other issues. The overlay is otherwise well-structured: pinned image, scoped allowed tools, outputStyle: Proactive, shared system prompt.

4. Five new advisory checker scripts (_shared/advisory_billing_*.py)

Severity: OK

High-quality structural gate scripts. Each validates flow topology, wiring, contract declarations, and anti-hardcode invariants that behavioral debug runs cannot see. Key observations:

• Every script is self-contained with no cross-_shared/ imports
• Error messages are diagnostic and actionable — they explain why the assertion matters, not just what failed
• The unwrap() helper is duplicated across all five scripts — minor DRY concern but acceptable given the self-containment constraint on checker scripts
• Each script exits cleanly via sys.exit(f"FAIL: …") on failure, print("OK: …") on success

The scripts cover genuinely distinct structural properties that no behavioral rung can see (e.g., mutual unreachability of parallel branches, context-handle wiring, connection binding real-key validation, IxP fileRef spelling).

5. _shared/check_bindings_no_stubs.py (new)

Severity: OK

Validates that generated bindings.json or bindings_v2.json files carry non-stub resource keys. Handles both the bindings and resources schema variants. The EXCLUDED_PARTS set prevents false matches on scaffolds, fixtures, and node_modules.

6. _shared/check_inline_agent.py

Severity: OK

Default glob pattern changes from */*/*/agent.json to **/agent.json with recursive=True. This makes the checker robust to varying directory depths.

7. connector_features/drive_to_slack.yaml

Severity: Low — prompt scalar style inconsistency

The initial_prompt uses a double-quoted string with embedded \n (line 23) while all other task YAMLs in this PR use | block scalars. Functionally equivalent but inconsistent. Previously flagged; author responded that the decoded prompt bytes are the contract and are intentionally unchanged.

8. connector_features/paginated_reference_lookup.yaml

Severity: OK

Whitespace normalization (inline tags → block list, indentation). Criteria preserved with identical weights, patterns, and thresholds. Added feature:records tag. No functional change.

9. connector_features/ceql_where.yaml

Severity: OK

Minor prompt wording change: failure branch now says "CeqlWhere test failed" (previously just "stop the flow"). Clearer intent for the agent.

10. connector_features/complex_array.yaml

Severity: OK

Connector switch from Slack (uipath-salesforce-slack Create Group DM) to Act! 365 (uipath-act-act365 add-activity). Adds Decision + Terminate nodes to the criteria. Removes require_success: true from the validate criterion. Weight adjustments are reasonable — the connector-key presence check is now the primary structural gate (weight 3.0). The uipath-salesforce-slack tag is correctly removed from tags.

11. ixp/e2e_01_invoice_extraction_greenfield.yaml

Severity: OK

Drops 3 criteria (5.5 weight points): skill_triggered (2.0), IxP registry search command_executed (2.0), SharePoint registry search command_executed (1.5). The remaining 5 criteria (flow validate, IxP node presence, SharePoint presence, HTTP node, $vars reference) form the structural intersection. The description field documents the validate-only rationale. Adds run_limits at the bottom (was previously inherited).

12. Five multi_node/billing_* task YAMLs

Severity: OK

All five follow the same pattern:

• Retain v1-base criteria (validate + domain checker) with original weights
• Add 1–2 report-only advisory criteria (pass_threshold: 0.0, weight: 1.0) pointing to the new _shared/advisory_*.py scripts
• Remove "Do not run debug yourself" from prompts (loop-neutral)
• Two tasks (billing_dispute_analyst, billing_resolution_writer) raise task_timeout from 2400→3300
• Two tasks (billing_discrepancy_detector, billing_invoice_lookup) add a check_bindings_no_stubs.py advisory

Consistent contract across all five.

13. single_node/coded_agent/coded_agent.yaml

Severity: OK

Adds explicit max_turns: 200 and task_timeout: 2400 (previously inherited). Whitespace normalization on criteria and post_run. The "Do NOT ask for approval" suffix is added to the prompt. Clean.

14. Skill doc changes (4 files)

Severity: OK

These are correctness fixes to the Maestro Flow skill docs, bundled with the task alignment:

• CAPABILITY.md (line 137): Corrects the [*] array-field anti-pattern — previously said to use =js:() expressions, now says to pass plain arrays matching the field's dataType. Aligns with the connector/impl.md fix.
• connector/impl.md: Corrects the array-field authoring table (plain arrays instead of =js:() wrappers), removes the "connector output shape" advisory table (about list returning bare array vs retrieve returning object), and removes the duplicate "array field round-trips empty" troubleshooting row. These removals simplify guidance where the original was found to be incorrect or misleading.
• loop/impl.md: Removes the cross-reference to the now-removed connector output shape note. Clean follow-through.
• rpa/impl.md: Simplifies RPA discovery guidance — removes the brand-name search warning and "empty result is not authoritative" paragraph. The remaining guidance ("tell the user to create the RPA project") is sufficient.

What's Missing

• Nothing identified. The changes are well-scoped and internally consistent.

Area Ratings

Area	Status	Notes
Frontmatter	OK	All task YAMLs have valid structure
E2E Tests	OK	No new skills — test coverage N/A
Skill Body	OK	Skill doc fixes are correctness improvements
References & Assets	OK	New advisory scripts are thorough and self-contained
Repo Hygiene	Issue	Personal path /home/tmatup in overlay (acknowledged, tracked as coder_eval#100)

Issues for Manual Review

• eu.anthropic.claude-sonnet-5 model ID (line 48, overlay) — verify this is a valid Bedrock model identifier for the campaign's routing. Not verifiable from the repo.
• skills-image:sg1 docker image tag (line 13, overlay) — confirm the campaign's build pipeline tags images this way.
• Connector output shape table removal — the removed table in connector/impl.md (about list returning bare array vs retrieve returning object) was guidance agents used when wiring loop collections. Verify the removal doesn't cause regressions in connector-loop tasks outside this PR's scope. The matching cross-reference in loop/impl.md was also correctly removed.
• RPA discovery simplification — the removed "empty result is not authoritative" guard in rpa/impl.md previously prevented agents from prematurely concluding a process was unpublished. Verify RPA-workflow tasks don't regress.

Conclusion

Clean, well-structured alignment of nine task files to a shared evaluation contract. The latest commit's read-only .skills shadow mount is a correct approach to blocking auth-cache leakage via uip skills list. The Dockerfile's migration to BuildKit secrets is a meaningful security improvement. The advisory checker scripts are thorough and well-documented. The skill doc corrections (array-field authoring, RPA discovery simplification) are genuine fixes.

One previously-acknowledged issue remains: the hardcoded /home/tmatup target path in the overlay (tracked as coder_eval#100). No new issues introduced by the latest commit. Approve.

---

[tmatup]

Current head 5b8584be5 merges the latest skills main before final shakeout. The integrated task pair still passes the same-ground gate 9/9 and the shared checker suite 68/68.

@claude review

🤖 Generated with Claude Code
Co-Authored-By: Claude

[tmatup]

@bai-uipath @rockymadden The allowlisted path-to-GA approval is still required; the integration head is now 5b8584be5. Please review and approve this head when available.

🤖 Generated with Claude Code
Co-Authored-By: Claude

[tmatup]

Claude review triage (review completed on 3979fc089; integration head is now 5b8584be5):

• Medium — /home/tmatup target: no code change. coder-eval expands variables only on mount sources and forwards the host HOME into the container; /.uipath would make the pinned CLI miss auth. The portability defect remains tracked in coder-eval feat: Add meta-router skill for UiPath task dispatch #100. Paired Docker canaries prove this runner-specific target, the nested read-only shadow, and auth all work on both arms.
• Low — quoted prompt scalar: no code change. The decoded prompt bytes are the cross-repo contract and pass the sync gate; changing representation provides no runtime benefit and risks byte drift.
• Model and image manual checks: verified by paired canaries: both record the fully qualified Bedrock model in agent config/environment, raw model_used is identical, skills-image:sg1 launches, CLI 1.198.0 is present, and Proactive/plugin/auth delivery are green.
• Connector-output and RPA-doc removals: these arrived from current main and are not campaign-authored deltas. They remain owned by their already-merged changes; expanding this PR to revise them would violate the campaign scope fence.

Every severity and manual-review item is dispositioned; no prompt, criterion, checker, weight, or limit changed.

🤖 Generated with Claude Code
Co-Authored-By: Claude

[tmatup]

@claude review

[github-actions]

Claude finished @tmatup's task in 3m 41s —— View job

---

PR Review: Align Maestro Flow tasks for same-ground evaluation

• Read review criteria and project conventions
• Analyze full diff (21 changed files)
• Review CI workflow changes
• Review Dockerfile changes
• Review experiment overlay
• Review task YAML changes (9 tasks)
• Review advisory checker scripts (5 new + 1 bindings checker)
• Review shared checker changes
• Security and secrets scan
• Post findings

---

Summary

Aligns nine Maestro Flow task YAMLs to a shared same-ground evaluation contract (loop-neutral prompts, v1-base criterion weights, report-only structural advisories at pass_threshold: 0.0), adds a campaign-only experiment overlay, migrates the Docker build to BuildKit secrets for the npm credential, adds five structural advisory checker scripts plus a bindings stub detector, and cleans up dead imports. Head 5b8584be5 merges latest main; the merge touched only two unrelated connector feature files and introduced no conflicts with the PR's own changes.

Change-by-Change Review

1. .github/workflows/run-coder-eval.yml + .github/workflows/smoke-skills.yml

Severity: OK

Both workflows switch from --build-arg NPM_AUTH_TOKEN=… to --secret id=npm_auth_token,env=NPM_AUTH_TOKEN. The token is now exposed via an env: block and consumed as a BuildKit secret mount — it no longer appears in docker history or build output. Correct and a meaningful security improvement.

2. tests/docker/Dockerfile

Severity: OK

Well-structured rewrite. Key changes:

• # syntax=docker/dockerfile:1.7 enables BuildKit secret mounts (line 1)
• Stable numeric CLI releases install from public npm before the @uipath→GHP npmrc is written (lines 81–83), then prerelease/dev pins install from GHP (lines 90–92)
• The public_cli flag derivation (lines 76–80) correctly distinguishes 1.198.0 (stable → public) from 1.199.0-dev.7923 (prerelease → GHP) from dev (dist-tag → GHP)
• Credential arrives via --mount=type=secret,id=npm_auth_token,required=true — no ARG exposure (lines 61–65)
• LABEL org.uipath.cli.version added for image introspection (line 112)
• npmrc is still written and removed within the same RUN layer — no credential leak

3. tests/experiments/same-ground-headtohead.yaml

Severity: Medium — personal path (previously triaged)

Lines 42–43 hardcode /home/tmatup/.uipath as the mount target. Author has triaged: runner-specific by design (coder-eval forwards host HOME into Docker; /.uipath would make the CLI miss its auth cache). Tracked as coder_eval#100. The nested read-only .skills shadow mount (line 43) correctly blocks uip skills list from writing to the auth cache — Docker gives the more-specific child mount path precedence over the parent rw mount.

Otherwise well-structured: pinned skills-image:sg1, scoped allowed tools, outputStyle: Proactive, shared system prompt, single default variant.

4. Five advisory checker scripts (_shared/advisory_billing_*.py)

Severity: OK

High-quality structural gate scripts. Each validates flow topology, wiring, contract declarations, and anti-hardcode invariants that behavioral debug runs cannot observe. Observations:

• Every script is self-contained with no cross-module imports
• Error messages are diagnostic and actionable — they explain why each assertion matters, not just what failed
• The unwrap() helper is duplicated across all five scripts (minor DRY concern; acceptable given checker self-containment)
• fail() consistently uses sys.exit(f"FAIL: …") across all scripts
• The advisory_billing_discrepancy_detector.py graph-reachability check (lines 328–359) correctly verifies mutual unreachability of parallel branches — the one structural property v1's node-type check cannot see
• The advisory_billing_dispute_resolution.py end-node classification (lines 789–816) correctly distinguishes resolution paths (agent-sourced email body) from rejection paths (literal body) — a subtle invariant

5. _shared/check_bindings_no_stubs.py

Severity: OK

Schema-agnostic stub detector that handles both bindings and resources array formats. The EXCLUDED_PARTS set prevents false matches on scaffolds, fixtures, and node_modules. Uses assert for failures — consistent with how the file is invoked (as a run_command criterion where a non-zero exit code signals failure).

6. _shared/check_inline_agent.py

Severity: OK

Default glob changes from */*/*/agent.json to **/agent.json with recursive=True. Makes the checker robust to varying directory depths. Clean, minimal change.

7. connector_features/check_drive_to_slack.py

Severity: OK

Removes dead imports: os, sys.path manipulation, and from _shared.flow_check import run_debug. These were needed for the debug call that was removed in the structural-only scope — the checker now validates only flow structure and wiring, not runtime behavior. Clean.

8. connector_features/drive_to_slack.yaml

Severity: Low — prompt scalar style inconsistency

The initial_prompt uses a double-quoted string with embedded \n (line 23) while all other task YAMLs in this PR use | block scalars. Functionally equivalent. Author previously triaged: decoded prompt bytes are the contract and pass the sync gate; changing representation risks byte drift.

The criterion description correctly drops "and debug step finished green" (line 34) — consistent with the structural-only scope after removing the run_debug import.

9. connector_features/paginated_reference_lookup.yaml

Severity: OK

Whitespace normalization only (inline tags → block list, indentation alignment, blank-line removal). Criteria, weights, patterns, and thresholds are identical. Added feature:records tag. No functional change.

10. ixp/e2e_01_invoice_extraction_greenfield.yaml

Severity: OK

Drops 3 criteria totaling 5.5 weight points:

• skill_triggered (weight 2.0) — removed
• command_executed for IxP registry search (weight 2.0) — removed
• command_executed for SharePoint registry search (weight 1.5) — removed

The remaining 5 criteria (flow validate 3.0, IxP node presence 2.0, SharePoint presence 2.0, HTTP node 2.0, $vars reference 1.5) form the v1-base structural intersection as documented in the PR body. Description correctly documents the validate-only rationale. Adds the "Do NOT ask" suffix and explicit run_limits.

11. Five multi_node/billing_* task YAMLs

Severity: OK

All five follow the same pattern:

• Retain v1-base criteria (validate + domain checker) with original weights (3.0 + 5.0)
• Add 1–2 report-only advisory criteria (pass_threshold: 0.0, weight: 1.0) pointing to the new _shared/advisory_billing_*.py scripts
• Two tasks (billing_discrepancy_detector, billing_invoice_lookup) also add a check_bindings_no_stubs.py advisory
• Remove "Do not run debug yourself" from prompts (loop-neutral — agent can debug if useful)
• Consistent "Build the complete flow / without stopping to ask me." prompt suffix
• billing_dispute_analyst and billing_resolution_writer set task_timeout: 3300 (up from 2400)
• billing_dispute_resolution sets task_timeout: 5400, turn_timeout: 3600, max_turns: 200 — appropriate for the full-pipeline task

Contract is consistent across all five.

12. single_node/coded_agent/coded_agent.yaml

Severity: OK

Adds explicit max_turns: 200, turn_timeout: 1200, task_timeout: 2400 (previously inherited from experiment defaults). Adds expected_turns: 35. Adds the "Do NOT ask for approval" suffix. Whitespace normalization on criteria and post_run. The mode:build tag is absent but this predates this PR — not a regression.

What's Missing

Nothing identified. The changes are well-scoped and internally consistent. The integration merge from main introduced no conflicts.

Area Ratings

Area	Status	Notes
Frontmatter	OK	All task YAMLs have valid structure
E2E Tests	OK	No new skills — test coverage N/A
Skill Body	N/A	No skill SKILL.md files changed
References & Assets	OK	New advisory scripts are thorough and self-contained
Repo Hygiene	Issue	Personal path /home/tmatup in overlay (acknowledged, tracked as coder_eval#100)

Issues for Manual Review

• eu.anthropic.claude-sonnet-5 model ID (overlay line 48) — verify this is a valid Bedrock model identifier. Not verifiable from the repo.
• skills-image:sg1 docker image tag (overlay line 13) — confirm the campaign's build pipeline tags images this way.

Conclusion

Clean PR. The Dockerfile's migration to BuildKit secrets is a genuine security improvement — credentials no longer appear in build output or image history. The advisory checker scripts are thorough, well-documented, and self-contained. The task alignment to the same-ground contract is consistent across all nine tasks: loop-neutral prompts, v1-base criterion weights preserved, structural advisories report-only at pass_threshold: 0.0. The 5b8584be5 merge of main is clean (touched two unrelated connector files, no conflicts).

One previously-triaged issue remains: the hardcoded /home/tmatup target path in the overlay (tracked as coder_eval#100). No new issues introduced. Approve.

---

[tmatup]

Latest-head Claude review (5b8584be5) triage:

• Medium runner path: accepted only as the pinned-runner campaign overlay; coder-eval feat: Add meta-router skill for UiPath task dispatch #100 tracks destination-variable support. Paired isolated canaries verify auth and the nested shadow.
• Low scalar style: no edit because decoded prompt bytes are the fairness contract and sync 9/9.
• Manual model/image checks: paired canaries verify skills-image:sg1, Proactive, CLI 1.198.0, fully qualified Bedrock routing, and identical raw model_used.

No new findings; every severity is dispositioned without a task/checker/limit change.

🤖 Generated with Claude Code
Co-Authored-By: Claude

[tmatup]

@uipreliga The path-to-GA check still needs one allowlisted approval on integrated head 5b8584be5. Claude review and triage are complete; could you review and approve when available?

🤖 Generated with Claude Code
Co-Authored-By: Claude

[tmatup]

All nine Maestro Flow task shakeout rows have now executed on the current integrated branch stack. The latest head remains green on the same-ground gate (9/9), shared checker suite (68/68), latest-head Claude review, and all ordinary CI. All observed reds are retained and classified; no prompt, checker, weight, or limit was changed from run feedback.

path-to-ga approval is the sole merge gate. @bai-uipath @rockymadden @uipreliga — please provide the allowlisted approval when available.

🤖 Generated with Claude Code
Co-Authored-By: Claude

[tmatup]

@claude review

Please review the latest head. The only post-review change is a merge of current main touching the disjoint Maestro Case test surface; review all severities.

🤖 Generated with Claude Code
Co-Authored-By: Claude

[github-actions]

Claude finished @tmatup's task in 6m 56s —— View job

---

PR Review: Align Maestro Flow tasks for same-ground evaluation

• Read review criteria and project conventions
• Gather full diff and PR conversation
• Review CI workflow changes
• Review Dockerfile changes
• Review experiment overlay
• Review task YAML changes (9 tasks)
• Review new advisory checker scripts (5 files + 1 shared checker)
• Review shared checker modifications
• Check for secrets, hardcoded paths, cross-skill deps
• Compile findings and post review

---

Summary

Aligns nine Maestro Flow task YAMLs to a shared same-ground evaluation contract: loop-neutral prompts, v1-weighted intersection criteria, five new report-only structural advisory checkers at pass_threshold: 0.0, and a campaign-only experiment overlay. The Dockerfile migrates npm credentials from a build ARG to a BuildKit secret mount, and adds a split install path so stable CLI releases install from public npm while tool plugins continue from GitHub Packages.

Change-by-Change Review

1. .github/workflows/run-coder-eval.yml + .github/workflows/smoke-skills.yml

Severity: OK

Both workflows migrate from --build-arg NPM_AUTH_TOKEN=... to --secret id=npm_auth_token,env=NPM_AUTH_TOKEN. The token is set in the step's env: block (not expanded inline in the run: script), then BuildKit reads it from the env var and mounts it at /run/secrets/npm_auth_token. This prevents the credential from appearing in docker history or build logs. Correct and consistent across both files.

2. tests/docker/Dockerfile

Severity: OK

Key changes:

• Adds # syntax=docker/dockerfile:1.7 directive — required for RUN --mount=type=secret
• Removes ARG NPM_AUTH_TOKEN= (was leaking to image history)
• Reads credential from BuildKit secret mount at /run/secrets/npm_auth_token
• Adds a public/private CLI install split: stable semver pins (e.g. 1.198.0) install from registry.npmjs.org before the GHP npmrc is written; prerelease/dist-tag versions install from GitHub Packages after
• Adds LABEL org.uipath.cli.version for image introspection

The credential handling is sound — the npmrc is created and removed in a single RUN layer, and the token never appears as a build ARG. The case-based routing for public_cli correctly distinguishes stable semver ([0-9]* without -) from prerelease ([0-9]*-*) and dist-tags (*). Well-documented comments explain the rationale.

3. tests/experiments/same-ground-headtohead.yaml (new)

Severity: Low — runner-specific mount path (acknowledged)

The overlay is well-structured: pinned Docker image tag (sg1), Sonnet 5 model route, outputStyle: Proactive, scoped allowed tools, isolated auth home via env vars.

Lines 42–43 use ${SG_UIPATH_HOME}:/home/tmatup/.uipath:rw — the destination path contains a user-specific home directory. Per the prior review triage, this is intentional: coder-eval forwards HOME into the container and the CLI's auth cache requires a matching path. The portability gap is tracked as coder_eval #100. The source is parameterized via SG_UIPATH_HOME / SG_EMPTY_SKILLS env vars, so while the destination is runner-specific, the mount only activates when those vars are set.

4. Five new advisory checker scripts (_shared/advisory_billing_*.py)

Severity: OK

Five thorough structural gate scripts that validate flow topology, connector wiring, contract declarations, and anti-hardcode invariants. Each script:

• Has clear docstrings listing what it checks
• Uses fail() + sys.exit() with diagnostic messages explaining why each assertion matters
• Handles expression envelope unwrapping (unwrap()) correctly
• No secrets, no hardcoded personal paths

Code quality is high. The fail() and unwrap() helpers are duplicated across all five scripts (~5 copies each). A shared _lib module would reduce this, and check_bindings_no_stubs.py already has _lib in its EXCLUDED_PARTS set, suggesting shared infrastructure was considered. However, given the self-containment constraint on checker scripts and the advisory-only nature of these checks, the duplication is acceptable.

5. _shared/check_bindings_no_stubs.py (new)

Severity: OK

General-purpose checker that finds all bindings.json / bindings_v2.json files (excluding vendor/build dirs), loads them, and asserts that every binding carries a real (non-stub) resource key. Uses assert for error reporting (rather than the fail()/sys.exit() pattern used by the advisory scripts), which produces a Python traceback on failure instead of a clean FAIL: message — a minor inconsistency in diagnostic style, but functionally equivalent (both exit code 1).

6. _shared/check_inline_agent.py

Severity: OK

Default glob pattern changes from */*/*/agent.json to **/agent.json with recursive=True. Makes the checker robust to varying directory depths. Clean fix.

7. connector_features/check_drive_to_slack.py

Severity: OK

Removes unused run_debug import and os/sys.path manipulation. Dead code cleanup consistent with the removal of debug-based grading from the corresponding task.

8. connector_features/drive_to_slack.yaml

Severity: Low — style inconsistency

The initial_prompt is serialized as a double-quoted string with embedded \n and escaped inner quotes (line 23). Every other task YAML in this PR uses YAML block scalars (|). Functionally equivalent — the decoded prompt bytes are identical — but harder to read and edit. The prior review flagged this as Low; the author's triage notes the representation doesn't alter the agent input, so no change is expected.

Other changes are correct: criterion description drops "and debug step finished green" to match the structural-only scope; "Do NOT ask for approval" suffix added.

9. connector_features/paginated_reference_lookup.yaml

Severity: OK

Formatting normalization only: inline tags to block list, removed quotes, flattened indentation. All criteria preserved with identical weights, patterns, and thresholds. No functional change.

10. ixp/e2e_01_invoice_extraction_greenfield.yaml

Severity: OK

Drops three process-grading criteria (skill_triggered weight 2.0, two command_executed checks for registry searches weight 2.0 + 1.5 = 3.5) totaling the 5.5-point reduction described in the PR body. Remaining five criteria form the structural v1-base intersection. The description field now includes a rationale for the validate-only scope (lines 7–8), addressing the Theme 1 finding from the prior lint review. run_limits moved to bottom of file but values unchanged.

11. Five multi_node/billing_* task YAMLs

Severity: OK

All five follow a consistent pattern:

• Retain v1-base criteria (validate + domain checker) with original weights
• Add 1–2 advisory criteria (pass_threshold: 0.0, weight: 1.0) pointing to the new _shared/advisory_billing_*.py scripts
• Two tasks (billing_discrepancy_detector, billing_invoice_lookup) also add check_bindings_no_stubs.py advisory
• "Do not run debug yourself" removed from prompts (loop-neutral)
• billing_dispute_analyst and billing_resolution_writer raise task_timeout from 2400 → 3300
• billing_dispute_analyst and billing_resolution_writer drop uip agent init --inline-in-flow scaffolding from description/prompt

Advisory script references all resolve correctly. No secrets or hardcoded paths.

12. single_node/coded_agent/coded_agent.yaml

Severity: OK

Adds explicit max_turns: 200 and task_timeout: 2400 (previously inherited from experiment defaults). Makes the task self-documenting under the same-ground contract. expected_turns: 35 retained. Formatting normalization on criteria and post_run.

What's Missing

• mode:build tag on e2e_01_invoice_extraction_greenfield.yaml and coded_agent.yaml — test-writing rules require mode:* on every task. Pre-existing gap, not introduced by this PR.
• Nothing else identified.

Area Ratings

Area	Status	Notes
Frontmatter	OK	All task YAMLs have valid structure
E2E Tests	OK	No new skills — test coverage N/A
Skill Body	N/A	No skill SKILL.md files changed
References & Assets	OK	New advisory scripts are well-structured
Repo Hygiene	OK	Credential handling improved (ARG → BuildKit secret); prior Low findings triaged

Issues for Manual Review

• eu.anthropic.claude-sonnet-5 model identifier (same-ground-headtohead.yaml:48) — verify this is a valid Bedrock model ID. Not verifiable from repo alone.
• skills-image:sg1 docker image tag (same-ground-headtohead.yaml:13) — campaign-specific tag; confirm the build pipeline tags images accordingly.
• Advisory checker test fixture values — the forbidden literals / canonical answers in the advisory scripts (e.g. 1610, MCS-2026-04872, Enterprise) are test fixture values from the tenant. Verify they match current tenant seed data.

Conclusion

Clean, well-documented alignment of nine task files to a shared same-ground evaluation contract. The Dockerfile credential handling is a genuine security improvement (BuildKit secret mount instead of build ARG). The five advisory checker scripts are thorough and well-documented. The prior review's findings have been addressed or triaged with justification. Two Low-severity style items remain (quoted prompt scalar in drive_to_slack.yaml, runner-specific mount path in the overlay), both acknowledged and intentional. No blocking issues — approve.

[tmatup]

Latest-head Claude review (`a1ff8aca8`) triage:

• Low — runner-specific auth mount: accepted only in this campaign overlay. coder-eval feat: Add meta-router skill for UiPath task dispatch #100 tracks portable destination expansion; paired isolated canaries verify the pinned path, auth, and read-only catalog shadow on both arms.
• Low — quoted Drive prompt scalar: no edit. The decoded prompt bytes are the fairness contract and the 9/9 sync gate proves them equal; representation has no runtime effect.
• Pre-existing mode tags: outside this PR and not a regression; no campaign expansion.
• Manual model/image checks: paired canaries verify the fully qualified Bedrock route, identical raw model_used, Proactive, skills-image:sg1, Claude Code 2.1.177, coder-eval 0.9.5, and uip 1.198.0.
• Fixture values: live tenant probes and the complete shakeout verify MCS-2026-04872, 1610, and Enterprise against the current seeded resources.

Every severity and manual-review item is dispositioned. No prompt, criterion, checker, weight, or limit changed. The allowlisted path-to-GA approval remains the only merge policy gate; @bai-uipath @rockymadden @uipreliga, please approve the current head when available.

🤖 Generated with Claude Code
Co-Authored-By: Claude

[tmatup]

@sherifmak @uisherif The path-to-GA check needs one allowlisted approval on current head `a1ff8aca8`. The latest Claude review and all severity triage are complete; all nine tasks have branch shakeout evidence and the final pinned-image canary is 5/5. Please review and approve when available.

🤖 Generated with Claude Code
Co-Authored-By: Claude