Skip to content

Break out runs and batches per type, removing ID-prefix inference #34

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
jordanalexmeyer wants to merge 5 commits into
base: jam/cli-workos-connect
Choose a base branch
from
Open

Break out runs and batches per type, removing ID-prefix inference #34

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
1563a52
Break out runs and batches per type, removing ID-prefix inference
jordanalexmeyer Aug 23, 2026
48071f4
Move typed batches inspection group into batches.go
jordanalexmeyer Aug 23, 2026
13d2836
Add edit detections create for async form detection runs
jordanalexmeyer Aug 24, 2026
997f240
Fail non-zero on unknown subcommands; harden evals stub and grader
jordanalexmeyer Aug 24, 2026
87a9b8d
Remove accidentally committed stub build artifact
jordanalexmeyer Aug 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 10 additions & 5 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,8 @@ and flag changes.
extend parse contract.pdf > contract.md

# run a workflow async; poll later
RUN=$(extend run doc.pdf --using workflow_abc -o id)
extend runs watch "$RUN"
RUN=$(extend workflows run doc.pdf --using workflow_abc -o id)
extend workflows runs watch "$RUN"

# filter JSON with jq
extend extract invoice.pdf --using ex_abc --jq '.output.value.invoice_id' -o raw
Expand All @@ -112,13 +112,18 @@ Inputs can be a local path (auto-uploads), a `file_xxx` ID, or an

## Commands

extract | classify | split | run <input> --using <id>
extract | classify | split <input> --using <id>
parse <input>
edit <input> --schema schema.json
workflows run <input> --using <id>
<action> batch <inputs>... [--files-from list.txt]

runs get | list | watch | cancel | delete | update
batches get | watch
extract|classify|split runs get | list | watch | cancel | delete
parse runs get | list | watch | delete
workflows runs get | list | watch | cancel | delete | update
edit runs get | watch | delete
edit detections create | get
<action> batches get | watch
files upload | list | get | delete | download

extractors | classifiers | splitters | workflows
Expand Down
87 changes: 60 additions & 27 deletions evals/evals.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@
},
{
"type": "extend_call",
"text": "agent ran `extend extract` (not parse \u2014 totals are typed fields)",
"text": "agent ran `extend extract` (not parse — totals are typed fields)",
"must_contain": [
{
"argv_prefix": [
Expand Down Expand Up @@ -171,7 +171,7 @@
"contract.pdf"
],
"expected_output": "Agent picks `classify`, discovers a classifier ID via list rather than fabricating one. The prompt deliberately omits the file path; the agent should glance at the working directory.",
"notes": "Known hard case: the description triggers semantically (Claude responds 'to classify' in 5/5 runs), but the agent's snap 'ask user for the file' decision fires before the Skill tool is invoked, so the body's cwd-discovery rule never loads. Cannot be fixed from the SKILL.md alone \u2014 needs either Claude Code agent-loop changes or unusually aggressive description prose. Honest expected pass rate is ~0% with current Claude Code; kept in roster as a measurement of this limitation.",
"notes": "Known hard case: the description triggers semantically (Claude responds 'to classify' in 5/5 runs), but the agent's snap 'ask user for the file' decision fires before the Skill tool is invoked, so the body's cwd-discovery rule never loads. Cannot be fixed from the SKILL.md alone — needs either Claude Code agent-loop changes or unusually aggressive description prose. Honest expected pass rate is ~0% with current Claude Code; kept in roster as a measurement of this limitation.",
"expectations": [
{
"type": "skill_activates",
Expand Down Expand Up @@ -282,10 +282,11 @@
},
{
"type": "extend_call",
"text": "agent ran `extend run` with --using",
"text": "agent ran `extend workflows run` with --using",
"must_contain": [
{
"argv_prefix": [
"workflows",
"run"
],
"flag": "using"
Expand Down Expand Up @@ -401,7 +402,7 @@
"id": "S-2",
"category": "single-command",
"path": "B",
"prompt": "Start the extract on invoice.pdf using ex_invoiceQ3 \u2014 but don't block. Just give me the run ID so I can check on it later.",
"prompt": "Start the extract on invoice.pdf using ex_invoiceQ3 — but don't block. Just give me the run ID so I can check on it later.",
"files": [
"invoice.pdf"
],
Expand Down Expand Up @@ -439,14 +440,15 @@
"path": "B",
"prompt": "Watch run exr_demo_processed until it finishes and tell me the result.",
"files": [],
"expected_output": "Agent runs `extend runs watch exr_demo_processed`.",
"expected_output": "Agent runs `extend extract runs watch exr_demo_processed`.",
"expectations": [
{
"type": "extend_call",
"text": "agent ran `extend runs watch exr_demo_processed`",
"text": "agent ran `extend extract runs watch exr_demo_processed`",
"must_contain": [
{
"argv_prefix": [
"extract",
"runs",
"watch"
],
Expand All @@ -469,14 +471,15 @@
"path": "B",
"prompt": "What's the result of run exr_demo_processed?",
"files": [],
"expected_output": "Agent runs `extend runs get exr_demo_processed`.",
"expected_output": "Agent runs `extend extract runs get exr_demo_processed`.",
"expectations": [
{
"type": "extend_call",
"text": "agent ran `extend runs get exr_demo_processed`",
"text": "agent ran `extend extract runs get exr_demo_processed`",
"must_contain": [
{
"argv_prefix": [
"extract",
"runs",
"get"
],
Expand Down Expand Up @@ -598,14 +601,15 @@
"path": "B",
"prompt": "Get me JSON for run exr_demo_processed with just the status and processedAt fields.",
"files": [],
"expected_output": "Agent runs `extend runs get exr_demo_processed -o json --jq '{status, processedAt}'`. Does NOT combine -o markdown with --jq.",
"expected_output": "Agent runs `extend extract runs get exr_demo_processed -o json --jq '{status, processedAt}'` (or equivalent jq filter). Does NOT combine -o markdown with --jq.",
"expectations": [
{
"type": "extend_call",
"text": "agent ran runs get with -o json and --jq",
"must_contain": [
{
"argv_prefix": [
"extract",
"runs",
"get"
],
Expand All @@ -622,6 +626,7 @@
"must_not_contain": [
{
"argv_prefix": [
"extract",
"runs",
"get"
],
Expand All @@ -646,14 +651,15 @@
"path": "B",
"prompt": "Print run exr_demo_processed as plain markdown so I can paste it into our Slack thread.",
"files": [],
"expected_output": "Agent runs `extend runs get exr_demo_processed -o markdown`. Does NOT pass --jq.",
"expected_output": "Agent runs `extend extract runs get exr_demo_processed -o markdown`. Does NOT pass --jq.",
"expectations": [
{
"type": "extend_call",
"text": "agent ran runs get with -o markdown",
"must_contain": [
{
"argv_prefix": [
"extract",
"runs",
"get"
],
Expand All @@ -669,6 +675,7 @@
"must_not_contain": [
{
"argv_prefix": [
"extract",
"runs",
"get"
],
Expand Down Expand Up @@ -730,10 +737,11 @@
},
{
"type": "extend_call",
"text": "agent queried failures with runs list",
"text": "agent queried failures with extract runs list",
"must_contain": [
{
"argv_prefix": [
"extract",
"runs",
"list"
],
Expand All @@ -755,7 +763,7 @@
"invoice.pdf",
"extractor.json"
],
"expected_output": "Agent issues create \u2192 versions create \u2192 extract in roughly that order.",
"expected_output": "Agent issues create → versions create → extract in roughly that order.",
"expectations": [
{
"type": "extend_call",
Expand Down Expand Up @@ -1027,6 +1035,7 @@
"must_contain": [
{
"argv_prefix": [
"workflows",
"run"
],
"args": [
Expand All @@ -1041,6 +1050,7 @@
"must_not_contain": [
{
"argv_prefix": [
"workflows",
"run"
],
"flag_value": {
Expand Down Expand Up @@ -1114,6 +1124,7 @@
"must_not_contain": [
{
"argv_prefix": [
"workflows",
"run"
]
}
Expand Down Expand Up @@ -1209,6 +1220,7 @@
"must_not_contain": [
{
"argv_prefix": [
"workflows",
"run"
]
}
Expand Down Expand Up @@ -1237,7 +1249,7 @@
"path": "B",
"prompt": "List all my edit runs from this week.",
"files": [],
"expected_output": "Agent recognizes edit runs are not listable (CLI rejects --type edit). Suggests `extend runs get edr_xxx` for individual edit runs.",
"expected_output": "Agent recognizes edit runs are not listable (there is no `edit runs list` command). Suggests `extend edit runs get edr_xxx` for individual edit runs.",
"expectations": [
{
"type": "stable_answer",
Expand All @@ -1248,7 +1260,7 @@
"type": "extend_call",
"text": "agent did NOT loop on the rejected command (max 2 attempts)",
"count_under": {
"runs.list": 3
"edit.runs.list": 3
}
}
]
Expand All @@ -1259,7 +1271,7 @@
"path": "B",
"prompt": "Cancel parse run pr_demo_processed for me.",
"files": [],
"expected_output": "Agent recognizes parse runs cannot be cancelled \u2014 explains rather than retrying indefinitely.",
"expected_output": "Agent recognizes parse runs cannot be cancelled — explains rather than retrying indefinitely.",
"expectations": [
{
"type": "stable_answer",
Expand All @@ -1279,32 +1291,54 @@
"id": "Q-3",
"category": "run-type-quirk",
"path": "B",
"prompt": "Show me workflow batch wfb_xxx \u2014 I want to know what's in it.",
"prompt": "Show me workflow batch wfb_xxx — I want to know what's in it.",
"files": [],
"expected_output": "Agent uses `extend runs list --type workflow --batch wfb_xxx`. Does NOT use `extend batches get`.",
"expected_output": "Agent uses `extend workflows runs list --batch wfb_xxx`. Does NOT use a typed `batches get` (workflow batches have no GET endpoint).",
"expectations": [
{
"type": "extend_call",
"text": "agent used `extend runs list --type workflow --batch`",
"text": "agent used `extend workflows runs list --batch`",
"must_contain": [
{
"argv_prefix": [
"workflows",
"runs",
"list"
],
"args": [
"--type",
"workflow"
"--batch"
]
}
]
},
{
"type": "extend_call",
"text": "agent did NOT use `extend batches get`",
"text": "agent did NOT use a typed `batches get` on a workflow batch",
"must_not_contain": [
{
"argv_prefix": [
"extract",
"batches",
"get"
]
},
{
"argv_prefix": [
"parse",
"batches",
"get"
]
},
{
"argv_prefix": [
"classify",
"batches",
"get"
]
},
{
"argv_prefix": [
"split",
"batches",
"get"
]
Expand All @@ -1319,24 +1353,23 @@
"path": "B",
"prompt": "Show me every FAILED extract run from the last 30 days. List them with their failure reason.",
"files": [],
"expected_output": "Agent uses `extend runs list --type extract --status FAILED` and iterates pages with --page-token (NOT --all).",
"expected_output": "Agent uses `extend extract runs list --status FAILED` and iterates pages with --page-token (NOT --all).",
"stub_config": {
"default_mode": "paginated",
"pages": 2
},
"expectations": [
{
"type": "extend_call",
"text": "agent issued `extend runs list --type extract --status FAILED`",
"text": "agent issued `extend extract runs list --status FAILED`",
"must_contain": [
{
"argv_prefix": [
"extract",
"runs",
"list"
],
"args": [
"--type",
"extract",
"--status",
"FAILED"
]
Expand All @@ -1354,9 +1387,9 @@
},
{
"type": "extend_call",
"text": "agent paginated (2+ runs list calls)",
"text": "agent paginated (2+ extract runs list calls)",
"count_at_least": {
"runs.list": 2
"extract.runs.list": 2
}
},
{
Expand Down
2 changes: 1 addition & 1 deletion evals/runner/grade/extend_call.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ func anyCallMatches(calls []CallRecord, p spec.ExtendCallPredicate) bool {
}

// countCalls counts recorded calls whose positional verb path begins
// with the given dotted path (e.g. "extract" or "runs.list").
// with the given dotted path (e.g. "extract" or "extract.runs.list").
func countCalls(calls []CallRecord, verbPath string) int {
want := strings.Split(verbPath, ".")
if verbPath == "" {
Expand Down
4 changes: 4 additions & 0 deletions evals/runner/grade/fabrication.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@ var DefaultFabricationPatterns = []string{
`splr_[a-zA-Z0-9_-]+`,
`pr_[a-zA-Z0-9_-]+`,
`edr_[a-zA-Z0-9_-]+`,
`sgr_[a-zA-Z0-9_-]+`,
`bpr_[a-zA-Z0-9_-]+`,
`bpar_[a-zA-Z0-9_-]+`,
`batch_[a-zA-Z0-9_-]+`,
`workflow_[a-zA-Z0-9_-]+`,
`workflow_run_[a-zA-Z0-9_-]+`,
`file_[a-zA-Z0-9_-]+`,
Expand Down
Loading