test(process-mining): cover model editing, publish, query info and source templates

[Cellcote]

Part 3 of the process-mining eval-set work, after #2545 (grader fix) and #2546
(linter fix). Independent of both — merges in any order.

Why

Auditing the process-mining eval set while chasing the flaky nightly turned up
two things beyond the grader bug:

• 20 of 44 uip pm verbs were referenced by any criterion.
• Three Critical Rules had no task at all, including one that is named in
  the skill's own frontmatter.

Rule	Before
6 — --wait on async commands	never asserted anywhere
8 — dev subset → publish	no task; apps publish untested
10 — apps model fields / DNA-46960	zero coverage

Rule 10 was the worst: references/model-editing.md is a full reference, the
numeric→duration lockout is called out in both description and when_to_use,
and nothing exercised it.

The five tasks

model_fields_data_kind — integration

Rule 10 and the DNA-46960 footgun. A throughput field is numeric; a metric
compares it to a numeric constant; the user wants duration.

Flipping the kind alone is the bug — that is exactly how the customer app
reached the state where no dashboard opens ("Must be duration, not numeric, for
the 'lt' input"). So the task grades fields set --kind duration and a
follow-up --expression edit, plus the pushed expression's constant via
file_contains on a fixed path (the pattern add_table_queryable already uses
for table.json). Negatives cover kinds outside the settable enum and the
structural id / ref kinds.

data_model_inspect_smoke — smoke

apps data-model get. Foreign keys live only in the structural model, not
the semantic one — the exact namespace confusion that produced the
apps model add-table grader bug fixed in #2545. Pins the read half of the
add-table workflow; add_table_queryable pins the write. Negatives guard
against mutating on a read-only request.

query_info_percentile — smoke

Rule 7's mechanism rather than its sugar. query_group_by_sugar covers the
path where --group-by/--metric resolves ids for the agent; percentile has no
such shorthand — it takes --field <id>, so discovery via query info cannot be
skipped without hitting UserError_FieldNotFound. Also pins that --values are
fractions in 0..1, and that --stage published is a dead end.

publish_promote_to_dashboards — integration

Rule 8 and the publish loop. apps publish promotes the definition, not the
rows: the result carries IngestionNeeded: true and published data is stale
until a re-ingest. Also the first task anywhere to assert --wait (Rule 6), and
it pins the documented CLI caveat that query --stage published is unreachable
(UserError_InvalidOrNoIngestion) — an agent that "verifies" the promotion that
way has followed a dead end instead of pointing the user at the dashboards.

source_template_selection — integration

Rule 2's untested half. custom_app_pipeline_e2e covers a flat log choosing
uipath.custom; nothing covered the other branch. A real multi-table SAP P2P
extract must choose uipath.p2p.sap — reaching for uipath.custom throws away
the template's entire extraction and event-log construction layer. Also pins
where a source template's expected input tables come from (the generated
transformations / sources.yml, not the model object) and that the
Cases.sql gotcha is uipath.custom-only.

Coverage delta

20/44 → 31/44 verbs. Newly exercised: app-types get,
apps data-model get, apps model fields list, apps model fields set,
apps publish, query info, query percentile.

Still uncovered, deliberately — these need a live tenant with real data to grade
anything meaningful, so command-construction tasks would only assert that a verb
was typed: query rca / insights / layout / details, transformations run
/ status / logs, ingestions logs, apps model get / update,
apps model fields remove.

Conventions

• Tags: skill + tier + mode:* + lifecycle:*, closed vocabularies only.
• No agent: or sandbox: blocks — inherited from the experiment. No
  env_packages.
• Command-construction, matching the existing process-mining tasks: no live
  tenant, graded on which commands the agent chooses, with the same "commands
  will fail with auth errors, run each exactly once" prompt caveat.
• Graded on side effects only — command_executed, command_not_executed,
  file_contains, skill_triggered. No criterion gates on --output json.

Verification

• All 5 parse; task ids unique across the 11 tasks in the directory.
• Tag check against the closed vocabularies in tests/README.md — all clean.
• CLI-verb reachability with the patched checker from fix(tests): make the CLI-verb linter see leaf verbs and negative criteria #2546: 0 High, 0 Medium
  on the new tasks. (Running it over the whole directory reports 1 High on
  add_table_queryable.yaml — that is the bug fix(process-mining): grade the real add-table verb and fix required tags #2545 fixes, not something
  introduced here.)
• Weights follow the guidance in .claude/rules/test-writing.md: 3.0 primary
  validation, 2.0–2.5 core command/artifact content, 1.0–1.5 supporting.

Passing-run claim: these are new tasks and have not been run end-to-end
against a tenant — I do not have one wired up here. They are validated
statically (parse, tags, verb reachability against catalog 1.200.0-dev.8173) and
every graded verb and flag is taken from the skill's own references. Someone with
tenant access should run the set once before merge; expect the first run to
retune weights or drop a criterion rather than pass clean, particularly
model_fields_data_kind, which is the most demanding of the five.

🤖 Generated with Claude Code

[github-actions]

Claude finished @Cellcote's task in 3m 33s —— View job

---

Coder-eval task lint (advisory)

• Read lint rubric
• Identify changed task YAMLs (5 files)
• Check PR body for passing-run evidence
• Read all changed tasks and 6 sibling tasks
• Apply rubric to each task
• Within-PR duplicate check
• Post final lint results

---

Coder-eval task lint (advisory)

5 task YAMLs changed; verdict counts: 0 Critical, 0 High, 0 Medium, 0 Low, 5 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

❌ High — PR body explicitly states: "Passing-run claim: these are new tasks and have not been run end-to-end against a tenant — I do not have one wired up here." The author has done thorough static validation (parse, tags, verb reachability) and is transparent about the gap, but the convention requires a passing-run claim. Please edit the PR description to add a claim once someone with tenant access runs the set, e.g.: Ran <task-id> locally and it passed.

Per-task lint

tests/tasks/uipath-process-mining/data_model_inspect_smoke.yaml — verdict: OK

Well-scoped smoke. Goal-oriented prompt ("show me which tables are registered … and how each one links back to Cases"). Primary criterion validates the correct namespace choice (apps data-model get, not apps model get). Two negatives guard against mutation on a read-only request. Complements add_table_queryable (write side) without overlap.

tests/tasks/uipath-process-mining/model_fields_data_kind.yaml — verdict: OK

Strong integration task covering Rule 10 and the DNA-46960 footgun. Five positive criteria (discovery → kind change → expression edit → artifact content check) and two negatives (invalid kinds, structural kinds). The expression.json artifact check (line 78–85) is a legitimate ground-truth anchor — it inspects the actual CLI payload, not a self-report. Weight distribution follows guidance: 3.0 primary, 2.5 core, 1.0–1.5 supporting.

tests/tasks/uipath-process-mining/publish_promote_to_dashboards.yaml — verdict: OK

First task anywhere to assert --wait (Rule 6) and the only task exercising apps publish (Rule 8). Good separation of concerns: three positive criteria (publish → re-ingest → --wait) and two negatives (querying published stage, hand-rolling a poll loop). The [\s\S]* between create and --wait (line 66) is appropriate for matching flags within one tool invocation.

tests/tasks/uipath-process-mining/query_info_percentile.yaml — verdict: OK

Complements query_group_by_sugar by testing the mechanism (raw field ids via query info) rather than the sugar. The --values 0?\.\d pattern (line 65) correctly validates that percentile points are fractions in 0..1, not integers. Four criteria cover: discovery → query construction → value domain → dead-end guard.

tests/tasks/uipath-process-mining/source_template_selection.yaml — verdict: OK

Tests the "other branch" of Rule 2 — choosing uipath.p2p.sap for a multi-table SAP extract instead of uipath.custom for a flat log. The 3.0-weight negative on --type uipath.custom (line 86–89) is the key differentiator from custom_app_pipeline_e2e. Also pins where input tables come from (transformations, not model), adding coverage for transformations (list|get) and app-types get.

Within-PR duplicates

No duplicate clusters detected. Each task exercises materially distinct CLI verbs and skill rules:

Task	Primary verbs	Skill rule
data_model_inspect_smoke	apps data-model get	structural vs. semantic namespace
model_fields_data_kind	apps model fields list, fields set	Rule 10 (DNA-46960)
publish_promote_to_dashboards	apps publish, ingestions create --wait	Rules 6, 8
query_info_percentile	query info, query percentile	Rule 7 mechanism
source_template_selection	app-types list/get, apps create, transformations list/get	Rule 2

Conclusion

✅ All 5 changed tasks pass the rubric — well-scoped, goal-oriented prompts, strong criteria with both positive and negative assertions, no self-report anti-patterns, no gameability concerns, no duplicates.

⚠ Evidence of passing run is missing (High, advisory). The author is transparent about this and explicitly asks for someone with tenant access to run the set before merge. Static validation (parse, tags, verb reachability) is confirmed clean.

Note: Could not run scripts/check-cli-verbs.py due to sandbox permissions. PR body reports 0 High, 0 Medium on the new tasks from the patched checker.

---

[github-actions]

Claude finished @Cellcote's task in 3m 29s —— View job

---

PR Review: test(process-mining) — model editing, publish, query info, source templates

• Read review criteria and project conventions
• Read full diff and all changed files
• Validate task YAML structure and tags
• Check for cross-skill references and secrets
• Post review findings

---

Summary

Adds five new coder_eval task YAMLs for the uipath-process-mining skill, covering previously untested critical rules (6, 8, 10) and CLI verbs (apps data-model get, apps model fields list/set, apps publish, query info, query percentile, app-types get). All five are command-construction tasks (no live tenant), consistent with the existing process-mining test pattern.

Change-by-Change Review

1. tests/tasks/uipath-process-mining/data_model_inspect_smoke.yaml

Severity: OK

Clean smoke test for the structural-vs-semantic data model namespace distinction. Required tags present (skill, tier:smoke, mode:diagnose, lifecycle:discover). Task ID skill-pm-data-model-inspect-smoke follows the convention. Two command_not_executed negatives guard against mutation on a read-only request. Weights are appropriate (3.0 primary, 2.0/1.5 negatives, 1.0 skill trigger).

2. tests/tasks/uipath-process-mining/model_fields_data_kind.yaml

Severity: OK

Thorough integration test for Critical Rule 10 (DNA-46960). Grades both the --kind duration flip and the follow-up --expression edit — correctly encoding that flipping the kind alone is the bug, not the fix. The file_contains criterion on expression.json (line 79–85) validates content beyond just command shape, which is a good practice. Two command_not_executed negatives guard against invalid kinds and structural kinds.

3. tests/tasks/uipath-process-mining/publish_promote_to_dashboards.yaml

Severity: OK

First task anywhere to assert --wait (Rule 6). The criterion at line 65–69 correctly requires --wait on the same ingestions create invocation via [\s\S]*. The negative at line 71–75 against query --stage published pins the documented CLI caveat. The anti-poll-loop negative at line 77–81 is a creative guard. Weights are well-calibrated.

4. tests/tasks/uipath-process-mining/query_info_percentile.yaml

Severity: Low

Correctly tests the discovery path (query info) before query percentile --field <id>. The --values 0?\.\d pattern (line 65) validates fractions in 0..1 rather than percentages. The --stage published negative is consistent with publish_promote_to_dashboards.yaml.

One minor note: this task is tagged smoke but sets explicit turn_timeout: 600 and max_turns: 25 (line 22–24). The existing smoke tests (app_types_discovery_smoke.yaml) only set expected_turns and inherit timeouts. The explicit overrides aren't wrong — the task does require more steps than a simple discovery smoke — but it sits at the upper end of what "smoke" typically implies. Consider whether integration is a better fit, or drop turn_timeout and let the experiment default handle it.

5. tests/tasks/uipath-process-mining/source_template_selection.yaml

Severity: OK

Covers Rule 2's untested branch (multi-table SAP P2P extract → uipath.p2p.sap). The 3.0-weight negative at line 86–89 against --type uipath.custom is the most important criterion — reaching for uipath.custom on a structured source-system extract is the exact mistake this task exists to catch. The Cases.sql negative (line 91–95) is a good follow-on guard.

What's Missing

• No passing-run claim: The PR description transparently states these have not been run end-to-end against a tenant. The lint workflow flags missing claims as High severity. The author recommends someone with tenant access run the set before merge — this is the right call, especially for model_fields_data_kind, which is the most demanding task.
• Domain-specific descriptive tags (e.g., data-model, model-editing, publish, query, percentile, app-types, source-template) are not in the closed vocabulary in tests/README.md. This is a pre-existing pattern across all process-mining tasks (the existing add_table_queryable.yaml uses data-model, add-table; query_group_by_sugar.yaml uses query, aggregate; etc.), so it's not introduced by this PR. Worth a follow-up PR to either formalize these as a process-mining tag convention or remove them — but not a blocker here.

Area Ratings

Area	Status	Notes
Frontmatter	N/A	Task YAMLs, not skill definitions
E2E Tests	OK	3 smoke + 1 e2e already existed; this adds 2 smoke + 3 integration — good coverage expansion
Skill Body	N/A	No skill files changed
References & Assets	N/A	No references or assets changed
Repo Hygiene	OK	CODEOWNERS already covers tests/tasks/uipath-process-mining/. No secrets, no cross-skill refs, no binary files. Changes are scoped to the process-mining test directory

Issues for Manual Review

• Verb and flag accuracy: The graded CLI verbs (apps model fields set --kind, apps publish, query percentile --field, query info, apps data-model get, transformations list/get, app-types list/get) and their flags are taken from the skill's own references. A domain expert should verify these match the actual CLI behavior, especially the --expression flag on fields set and the --values fraction semantics on query percentile.
• End-to-end run: As the author notes, weights and criteria may need tuning after the first live run. model_fields_data_kind is the most likely candidate for adjustment given its multi-step grading (kind flip + expression rewrite + file content check).
• [\s\S]* bridging in patterns: Several command_pattern regexes use [\s\S]* to bridge between the verb and a flag (e.g., line 66 in publish_promote_to_dashboards.yaml, line 72 in source_template_selection.yaml). This works correctly if coder_eval matches each command invocation individually. If the pattern were ever matched against a concatenated transcript, it could false-positive across separate commands. The existing process-mining tasks use the same pattern, so this is validated, but worth noting for future maintainers.

Conclusion

This is a well-structured, well-documented PR that fills significant coverage gaps in the process-mining eval set (20/44 → 31/44 CLI verbs covered). The tasks are internally consistent, follow the established command-construction pattern, and the weight/threshold choices align with repo guidance. The only actionable item is the missing passing-run claim, which the author has flagged transparently. Approve once someone with tenant access confirms a first run (or accepts the static-validation-only claim for merge).

[Cellcote]

Passing-run claim — all five tasks green on first run

Ran locally with coder-eval (experiments/default.yaml, tempdir sandbox, claude-sonnet-5, 1 replicate), on this branch merged with #2545:

skill-pm-model-fields-data-kind              SUCCESS  1.000  turns=19
skill-pm-data-model-inspect-smoke            SUCCESS  1.000  turns=9
skill-pm-query-info-percentile               SUCCESS  1.000  turns=8
skill-pm-publish-promote-to-dashboards       SUCCESS  1.000  turns=11
skill-pm-source-template-selection           SUCCESS  1.000  turns=34

Full process-mining set in the same run: 10/11 SUCCESS. The one failure, skill-pm-query-group-by-sugar (0.750), is a pre-existing task untouched by this PR.

This replaces the "not run end-to-end" caveat in the description above. I expected model_fields_data_kind to need retuning; it passed clean, including the file_contains assertion on the pushed expression.

Caveats on how much this proves

1. CLI version. The run used @uipath/cli@1.198.0 (public npm latest). CI builds from @uipath/cli@dev on GitHub Packages — I could not install that (gh token lacks read:packages). 1.198.0 is missing apps data-model, apps model fields and apps publish, so three of these tasks ran against a CLI where the graded verb does not exist and still scored 1.000 — the agent typed the skill-taught command once despite the unknown-command error, which is what the command-construction design intends. On the @dev build those commands additionally resolve, so this is the pessimistic case.
2. One replicate, one model. claude-sonnet-5 locally; the nightly has been running gemini-3.5-flash and gpt-5.6-terra. Green here does not guarantee green under a different model — that is the same single-replicate weakness called out in fix(process-mining): grade the real add-table verb and fix required tags #2545.

Static checks, unchanged

• coder-eval plan — all 11 tasks valid.
• Tag vocabularies — clean.
• CLI-verb reachability with the checker from fix(tests): make the CLI-verb linter see leaf verbs and negative criteria #2546 — 0 High, 0 Medium on the new tasks.
• Pattern probe: 28 cases asserting each new command_pattern fires on the command the skill teaches and does not fire on near-misses (e.g. apps model get must not satisfy the structural data-model get criterion; --values 50,90 must not satisfy the fractional check). 0 wrong.