feat(uipath-insights): teach filter discovery and add filters smoke eval

[celestinryf]

Related to IN-13523.

What

Teaches the four filter discovery command groups landing in UiPath/cli#3449: filter-folders, filter-processes, filter-queues, filter-machines, each with a list subcommand. SKILL.md gets filter hooks in description/when_to_use, a Filter Discovery Commands section (four rules: no time flags, client-side --limit/--offset, empty is not proof of absence, results are permission-bounded) and a discover-before-you-act workflow; Critical Rule 1 is scoped to jobs commands only. references/investigation-playbook-guide.md routes folder-key lookups through filter-folders list first, with uip or folders list as the fallback. Adds tests/tasks/uipath-insights/filters/smoke.yaml and activation rows 026-029. The five jobs smoke tasks, the e2e tasks from #2551, and BASELINES_PCT are untouched.

Why

The commands ship in UiPath/cli#3449 (review approved, merge pending). Without skill guidance, agents guess folder keys or pass --time-range to commands that reject it; the discovery workflow makes them page through results before concluding a resource is absent. The backend applies a fixed 30-day activity window, which is why the skill frames empty results as "no recent activity", not "does not exist".

Scope

Skill content, one smoke task, four activation positives. E2e-depth coverage for these commands lives in the cli repo (scenarios/insights-tool/filter-discovery.md S1-S8 and filter-discovery.e2e.test.ts, both on cli#3449); this skill already has its e2e slice in #2551. The activation baseline (BASELINES_PCT) is deliberately not edited here; re-measuring recall with the new rows is the follow-up once a baseline entry exists for this skill.

Testing

• scripts/check-cli-verbs.py and scripts/check-task-driver.py both pass on the new smoke task, run 2026-08-10.
• The skill stays inside its size budgets. The frontmatter description is 492 characters against the 1024 cap. Description plus when_to_use is 1242 characters, and Claude Code truncates at 1536. SKILL.md is 316 lines, about 3.8k tokens against the 5k budget.
• The smoke task itself has not been run. It cannot pass yet; see Limitations.

Limitations

skill-insights-smoke-filter-discovery has no passing run yet, and cannot have one: the four commands exist only on cli#3449, and the PR smoke gate installs @uipath/cli@dev, which picks them up only when that PR merges to cli/main. The 2026-08-06 catalog snapshot (1.200.0-dev.8116) carries 0 filter verbs, consistent with that. A real passing-run claim will be added to this PR once the commands are on cli/main; this PR should not merge before that run exists.

Activation recall is not re-measured. The activation gate skips uipath-insights until a BASELINES_PCT entry exists, so rows 026-029 are inert for gating until the baseline lands.

How to Review

The Filter Discovery Commands section in SKILL.md carries the weight, especially the no-time-flags rule and the empty-is-not-absence rule, which are the two mistakes agents actually make with these commands. The playbook edits are routing, and the smoke YAML is a criteria list.

Checklist

• Tests added/updated for new logic — the filters smoke task; run evidence pending cli#3449 merge, see Limitations
• No secrets or credentials in code
• Commits follow conventional format
• PR ≤ 400 lines

CI note

The smoke gate (smoke-skills.yml) will fail this PR's filters task until cli#3449 merges, since the @uipath/cli@dev install cannot have the commands before then. The lint-tasks.yml advisory may also flag the missing passing-run claim; both resolve with the post-merge run described in Limitations.

[github-actions]

Claude finished @celestinryf's task in 3m 23s —— View job

---

PR Review: feat(uipath-insights): teach filter discovery and add filters smoke eval

• Read review criteria and project conventions
• Read full diff and changed files
• Validate skill structure and content
• Check test task compliance
• Verify links, cross-references, and security
• Post review findings

---

Summary

Adds four filter-* discovery command groups (filter-folders, filter-processes, filter-queues, filter-machines) to the uipath-insights skill — frontmatter, a new Filter Discovery Commands section with rules and a "discover before you act" workflow, playbook routing updates, one smoke task, and three activation rows. Well-scoped change that correctly separates the filter commands' semantics (no time flags, 30-day backend window, permission-bounded) from the existing jobs commands.

Change-by-Change Review

1. skills/uipath-insights/SKILL.md — +44/−5

OK

Frontmatter:

• description is ~492 chars (under 1024 cap). Combined with when_to_use is ~1242 chars (under 1536 truncation). Pass.
• when_to_use correctly adds filter-specific trigger phrases (which folders can you see, find the folder key, discover queues, filter-folders, filter-processes).
• Queue-item nuance is well-handled: "NOT for queue item metrics (queue discovery is supported via filter-queues; metrics are not)."

Body:

• Critical Rule 1 (line 50): scope correctly narrowed to jobs commands with a forward-ref to the new section. Clear.
• Filter Discovery Commands section (lines 166–201): four numbered rules cover the key behavioral differences. The "no time flags" and "empty is not proof of absence" rules directly address the two agent mistakes cited in the PR body. Prescriptive, numbered, copy-paste-ready commands.
• "Discover before you act" workflow (lines 184–201): step-by-step with pagination handling and explicit "never guess or fabricate a key" stop condition. Good anti-hallucination guardrail.
• Folder-key section (line 256): correctly offers both filter-folders list and uip or folders list as options with a clear reason to prefer one over the other for monitoring tasks.

One minor gap: the When to Use bullet list (lines 18–22) still only lists jobs-related scenarios. An agent that triggers on "find the folder key" and scans the "When to Use" section won't see discovery mentioned there. Since the frontmatter when_to_use handles activation and the Filter Discovery Commands section is clearly labeled, this is cosmetic — but adding a discovery bullet would improve in-skill navigation.

2. skills/uipath-insights/references/investigation-playbook-guide.md — +6/−3

OK

Playbook 6 (lines 109–114): correctly routes folder-key lookup through filter-folders list first, with uip or folders list as fallback. The comment "There the GUID is the folder's Key field (not FolderKey)" (line 114) clarifies the field-name difference between the two APIs — useful because an agent confusing Key with FolderKey would pass the wrong value.

Handoff table (line 147): updated row correctly prioritizes the in-skill command and keeps the cross-skill fallback. The row is long but table cells handle it fine.

No reference pointer discipline violations — the playbook routes to commands, doesn't restate the Filter Discovery Commands rules from SKILL.md.

3. tests/tasks/activation/uipath-insights.jsonl — +3/−0

OK

Rows 026–028 cover three distinct discovery intents: folder activity, folder-key lookup, and machine activity. Prompts are natural-language and don't name the skill. IDs follow the sequential numbering. The PR notes these are inert until a BASELINES_PCT entry exists — correctly documented in Limitations.

4. tests/tasks/uipath-insights/filters/smoke.yaml — +70/−0 (new file)

OK

• task_id: skill-insights-smoke-filter-discovery — follows skill-<domain>-<capability> pattern.
• tags: [uipath-insights, smoke, mode:build, lifecycle:discover] — all four required dimensions present, values are from the closed vocabulary.
• No sandbox or agent block — correctly inherits from experiment defaults (consistent with sibling tasks).
• No env_packages — correct per the test-writing rules.
• initial_prompt: goal-oriented, doesn't enumerate CLI flags. Includes pagination and error-handling instructions. Matches the skill's guidance.
• Success criteria:
  • skill_triggered (weight 1.5) — verifies activation.
  • Three command_executed checks (weight 2.0 each) — filter-folders, filter-processes, filter-machines.
  • command_not_executed (weight 1.5) — negative check that no time flags are passed to filter commands. This directly validates the "No time flags" rule.
  • Advisory --output json check (weight 1.0, pass_threshold 0) — correctly non-gating per the test-writing rules.

The prompt asks for folders, processes, and machines — intentionally excluding queues. This means filter-queues list has zero test coverage, but that's acceptable for a smoke test.

What's Missing

• "When to Use" body section lacks a discovery bullet. The when_to_use frontmatter handles activation, but the in-body "When to Use" section (SKILL.md:18–22) only lists jobs scenarios. Adding "Discovering which folders, processes, queues, or machines have recent activity for scoping" would help agents navigate once the skill is loaded. (Low — frontmatter activation is correct; body-section gap is navigational only.)
  Add discovery bullet →

• No filter-queues activation row. Rows 026–028 cover folders and machines but not queues. A prompt like "Show me which queues have been active recently" would test queue-discovery activation. (Low — the when_to_use field already includes discover queues, so activation should work; a row would verify it.)

Area Ratings

Area	Status	Notes
Frontmatter	OK	Description 492 chars, combined 1242 chars; → redirects present
E2E Tests	OK	Existing skill — no new e2e required for this change; PR description notes e2e slice lives in #2551
Skill Body	OK	Filter Discovery Commands section is prescriptive, numbered, copy-paste-ready
References & Assets	OK	Playbook routing updated, no duplicate content, no broken links
Repo Hygiene	OK	CODEOWNERS already covers both skill and test paths; no secrets; changes scoped to one skill

Issues for Manual Review

• CLI command correctness. The four filter-* commands are documented from cli#3449 which hasn't merged yet. The actual flag names (--limit, --offset), response shapes (FolderName, FolderKey, Pagination.Total, HasMore), and the 30-day activity window should be verified against the CLI implementation once it merges.
• filter-queues response shape. SKILL.md documents {QueueName, FolderKey} rows — confirm this matches the CLI's actual output schema.
• Pagination semantics. The skill says "--limit / --offset paginate client-side." Confirm whether these are truly client-side (CLI fetches all, then slices) or server-side (CLI passes to the API). This affects whether HasMore: false guarantees completeness.

Conclusion

Clean, well-scoped PR. The skill content is prescriptive and addresses the two concrete agent mistakes (adding time flags, treating empty as "does not exist"). The smoke task has solid criteria including the negative time-flag check. The two missing items (discovery bullet in "When to Use", queue activation row) are both Low and don't block merge. Recommend approve once the CLI commands land and the passing-run claim is added per the Limitations section.

---

|

[github-actions]

Claude finished @celestinryf's task in 2m 35s —— View job

---

Coder-eval task lint (advisory)

1 task YAML changed; verdicts: 0 Critical, 1 High, 0 Medium, 0 Low, 0 OK.

Rubric: .claude/commands/lint-task.md. This check is advisory and never blocks merge.

Evidence of passing run

❌ High — PR body explicitly states: "The smoke task itself has not been run; see Limitations" and "skill-insights-smoke-filter-discovery has no passing run yet, and cannot have one." The author documents a valid reason (commands exist only on cli#3449, not yet merged) and commits to adding a passing-run claim post-merge. Please edit the PR description to add a line like Ran skill-insights-smoke-filter-discovery locally and it passed. once cli#3449 merges and the task can run.

Per-task lint

tests/tasks/uipath-insights/filters/smoke.yaml — verdict: High

Issues:

• [High] CLI verb reachability (lines 35, 42, 49, 59, 67): filter-folders, filter-processes, and filter-machines verb paths do not exist in assets/uip-catalog-snapshot.json (catalog version 1.200.0-dev.8116). All five command_executed/command_not_executed criteria reference these verbs. The author documents this: commands ship in cli#3449 (merge pending); criteria become reachable when the nightly catalog refresh picks them up post-merge.
• [Low] Meaningful coverage (lines 25–70): All gating criteria are command_executed / command_not_executed — no output validation (json_check, run_command, file_contains). The command_not_executed for time flags (line 56) is a genuine correctness check (verifies the agent follows the skill's no-time-flags rule), and smoke tier on a live tenant with unpredictable results justifies lighter validation. Consistent with sibling smoke tasks in this skill.

Suggested fixes:

• No action needed on CLI verb reachability until cli#3449 merges and the catalog refreshes. Once it does, re-run python3 scripts/check-cli-verbs.py --json tests/tasks/uipath-insights/filters/smoke.yaml to confirm the verbs resolve.
• Consider adding one run_command or json_check criterion that validates the shape of filter command output (e.g., run_command with uip insights filter-folders list --output json and stdout_match checking for an array or []). This would upgrade coverage from "agent ran the right commands" to "commands returned parseable output". Low priority — the existing criteria are solid for smoke tier.

Within-PR duplicates

No duplicate clusters detected (only 1 task YAML added).

Conclusion

⚠ 1 task has issues, max severity High (CLI verb reachability — documented, resolves when cli#3449 merges). Advisory only — not blocking merge.

---

[celestinryf]

Addressing the two bot reviews, point by point (changes landed in 8aad100f0):

PR review findings

• "When to Use" lacks a discovery bullet — added: "Discovering which folders, processes, queues, or machines have recent activity, to pick an exact scope (see Filter Discovery Commands)".
• No filter-queues activation row — added row 029: "Show me which queues have been active recently."
• Smoke task excludes queues — this reviewed an earlier commit. The current task grades all four groups: filter-queues is in the prompt and has its own command_executed criterion, and expected_turns is 10.

Manual-review questions the review raised

• Pagination semantics — client-side, confirmed against the cli#3449 implementation: the backends return unpaginated arrays, and the CLI dedupes, sorts, then slices with --limit/--offset. So HasMore: false does mean the caller has seen every row the backend returned for the 30-day window.
• filter-queues response shape — {QueueName, FolderKey} is source-verified against the cli#3449 wrappers and its e2e suite. I'll re-check all four shapes against live --output json output in the same post-merge run that produces the passing-run claim.

Task lint

• High, verb reachability and no passing run — acknowledged; both resolve when cli#3449 merges and the catalog refreshes. I'll run the task, add the passing-run claim to the body, and re-run check-cli-verbs.py then. This PR should not merge before that.
• Low, add a run_command/json_check shape criterion — deliberately not adding for smoke tier: it would re-run live commands during grading (doubling live calls on the shared tenant), the sibling smoke tasks don't validate output shape either, and shape validation already lives in the cli repo's e2e suite (S1-S8). Happy to revisit as a separate integration-tier task if reviewers want graded shape coverage here.

[rliuup]

LGTM with nits.

[rliuup]

issue (non-blocking): This task already passed at weighted 1.00 on this head (smoke run reported 7/7 passed, status=SUCCESS) while cli#3449 is still open, because command_executed matches the command string and not its exit code — so all four criteria pass even when uip insights filter-* does not exist yet. Either soften the Limitations claim that a passing run is impossible before that merge, or add a run_command criterion (uip insights filter-folders list --output json, expected_exit_code: 0) so the task actually proves the CLI path.

[rliuup]

suggestion (non-blocking): Now that uip insights filter-queues list is a taught command, the existing negative criterion in tests/tasks/uipath-insights/smoke_critical_rules.yaml (command_pattern: 'uip\s+insights.*queue') can false-fail a weight-2.0 assertion if the agent runs filter-queues while explaining the queue-metrics boundary. Narrowing it to the actual intent — e.g. uip\s+insights\s+jobs[^&;|\n]*queue — keeps that guard meaningful.

[rliuup]

suggestion (non-blocking): The new discovery phrases overlap the uipath-platform activation set (uipath-platform-009 folders, -032 processes under a folder, -018 queue items), and the activation gate only re-runs skills whose own SKILL.md changed, so an activation lost there would only surface later on an unrelated uipath-platform PR. Worth running the uipath-platform activation dataset once before merge.

[rliuup]

suggestion (non-blocking): [^&;|\n]* also excludes newlines, so a backslash-continued invocation (uip insights filter-folders list \ then --time-range 60) slips past this guard — the exact mistake it exists to catch, and the jobs examples in SKILL.md use that continuation style. (?:[^&;|\n]|\\\n)* closes it without reintroducing the cross-command bleed.

[rliuup]

nit (non-blocking): mode:build does not match the taxonomy in tests/README.md (build = creating/designing/editing/deploying), and the five sibling uipath-insights tasks all use mode:diagnose; since the tag drives make tags and the evalboard, a read-only discovery task lands in the wrong bucket.

[rliuup]

nit (non-blocking): uip insights filter-* stays soft-stale in the verb gate even after a catalog refresh, since filter-* is never a real verb path. An inline <!-- uip-check-skip --> on this line clears the recurring warning.

[suranabhavya]

Ruilin's review covered the substantive points, fixing that post LGTM

[rohitjain-uipath]

+1 on Ruilin's comments. Lets fix them before merge, rest LGTM!