Prompt batch tester (acme-crm connector)

Runs 100+ prompts through Claude or Codex/ChatGPT CLI + your acme-crm MCP connector automatically, one fresh session per prompt, and records which tools fired and what the agent answered.

Claude is still the default because that is what the original script used:

python3 run_batch.py --agent claude --csv prompts.csv

To run the same CSV through Codex/ChatGPT CLI:

python3 run_batch.py --agent codex --csv prompts.csv

One-time setup

1. Install the CLI you want to test.

   For Claude Code (needs Node.js 18+):

   npm install -g @anthropic-ai/claude-code
   

   For Codex/ChatGPT CLI, install and log in using your normal Codex setup.

2. Log in once.

   Claude:

   claude
   

   Codex:

   codex
   

   Complete login, then quit (Ctrl-C / exit).

3. Put the files in an empty folder (not inside a code repo): run_batch.py and your prompts.csv.

4. Register + authenticate acme-crm. If you already added it in your terminal, skip the add line — just make sure it's authenticated.

   Claude:

   claude mcp add --transport http acme-crm https://YOUR-ACME-CRM-MCP-URL/mcp
   claude mcp login acme-crm
   

   Codex:

   codex mcp add acme-crm --url https://YOUR-ACME-CRM-MCP-URL/mcp
   codex mcp login acme-crm
   

   The login (OAuth) step is required because headless runs can't open the sign-in popup themselves. The token is cached afterward.

5. Verify:

   Claude:

   claude mcp list
   

   Codex:

   codex mcp list
   

   acme-crm should show as connected.

The connector name must match. The script expects acme-crm (see MCP_SERVER_NAME at the top of run_batch.py). If your connector is registered under a different name, change that one line.

Do I need the .mcp.json file?

No, if acme-crm is registered via the CLI's mcp add command (step 4) — headless runs pick it up automatically. The included .mcp.json is only a Claude fallback if you'd rather define the connector locally; if you use it, replace the placeholder URL. Codex uses its registered MCP config and ignores --mcp-config here.

Prepare your CSV

• One prompt per row. The script auto-detects the prompt column (looks for prompt, else uses the first column). Your file uses prompt — you're set.
• Optional followup column: if filled in for a row, that follow-up is sent in the same session after the first response. Leave blank for rows that need none.

Accepted input columns

Column	Required	What it does	Header names recognized (case-insensitive)
Prompt	Yes	The prompt sent in a fresh session.	prompt, prompts, input, query, message — if none are found, the first column is used.
Follow-up	No	Sent in the same session after the first reply. Blank = no follow-up for that row.	followup, follow_up, follow up, follow-up, followup_prompt, follow up prompt
Write	No	Marks a row as a write action, gating the fixed final prompt (see below). yes/no (also y/n, true/false, 1/0). Blank = treated as a write. If the column is absent entirely, every row counts as a write.	write, is_write, write_action, writes

Any other columns are ignored, so it's fine to keep notes/IDs/tags alongside. (With --auto-followup, the follow-up column is ignored — replies are generated.) Minimal file is just one prompt column:

prompt
Who does patrick superstar report to?
Who's out next week?

With follow-ups:

prompt,followup
Who's out next week?,Which of them are engineers?
How many leave days do I have left?,

Run

Pilot on the first 2 rows first:

python3 run_batch.py --csv prompts.csv --limit 2

Codex/ChatGPT CLI:

python3 run_batch.py --agent codex --csv prompts.csv --limit 2

Then the full run:

python3 run_batch.py --csv prompts.csv

Useful flags: --delay 3 (slow down), --model claude-sonnet-5 or another model name (pin a model for consistent evals), --start 40 (jump to a row), --timeout 600.

All flags

Flag	Default	What it does
--agent claude|codex	claude	Which CLI runner to use. codex is the Codex/ChatGPT CLI path.
--csv PATH	(required)	Input CSV of prompts.
--mcp-config PATH	.mcp.json	Claude-only local MCP config; ignored by Codex and ignored if you use a registered connector.
--outdir DIR	output	Where results.csv and raw logs are written.
--limit N	0 (all)	Only run the first N not-yet-done rows. Good for piloting.
--start N	0	Skip to this 0-based row index.
--delay SECS	2	Pause between calls (rate-limit friendliness).
--timeout SECS	600	Max seconds per prompt before giving up on that row.
--model NAME	claude-sonnet-5 for Claude, gpt-5.5 for Codex	Optional override. If omitted, the runner pins the agent-specific default model.
--truncate N	4000	Max characters of a response stored in the CSV (full text is always in the raw log).
--fresh	off	Archive any previous run in --outdir and start from row 0 (nothing deleted).
--auto-followup	off	Auto-decide and send follow-ups when a reply asks for input; ignores the CSV follow-up column.
--max-followups N	3	Auto mode only: cap on auto follow-up turns per prompt.
--final-prompt TEXT	(built-in text)	A fixed message sent as the last turn, after all follow-ups/auto-followups resolve, but only for rows marked as a write action (the Write column). Defaults to DEFAULT_FINAL_PROMPT in the script — edit it there instead of passing this. Pass "" to disable.
--codex-bypass-sandbox	off	Codex only: pass --dangerously-bypass-approvals-and-sandbox to codex exec. Use only in a trusted sandbox.

Model options for --model

Updated: 12 July 2026. Model availability can depend on your account, org policy, region/provider, and CLI version. If a model is blocked or unknown, open the model picker in the relevant CLI (/model) or check the official docs linked below.

Claude examples:

python3 run_batch.py --agent claude --model sonnet --csv prompts.csv
python3 run_batch.py --agent claude --model claude-sonnet-5 --csv prompts.csv

Claude --model value	What it means
default	Clears the override and uses your account/org default.
best	Uses Fable 5 if available, otherwise the latest Opus.
fable	Claude Fable 5 for the hardest/longest-running tasks.
opus	Latest Opus. On Anthropic API, currently Opus 4.8.
sonnet	Latest Sonnet. On Anthropic API, currently Sonnet 5.
haiku	Latest Haiku for fast/simple tasks.
opusplan	Opus during plan mode, then Sonnet for execution.
sonnet[1m]	Sonnet with long-context behavior where applicable.
opus[1m]	Opus with long-context behavior where applicable.
claude-fable-5	Full model ID for Claude Fable 5.
claude-opus-4-8	Full model ID for Claude Opus 4.8.
claude-sonnet-5	Full model ID for Claude Sonnet 5.
claude-haiku-4-5-20251001	Pinned Claude Haiku 4.5 snapshot.
claude-haiku-4-5	Claude Haiku 4.5 alias.

Codex/ChatGPT CLI examples:

python3 run_batch.py --agent codex --model gpt-5.6-sol --csv prompts.csv
python3 run_batch.py --agent codex --model gpt-5.6-terra --csv prompts.csv
python3 run_batch.py --agent codex --model gpt-5.5 --csv prompts.csv

Codex --model value	What it means
gpt-5.6-sol	OpenAI flagship model for complex reasoning and coding.
gpt-5.6	Alias for gpt-5.6-sol.
gpt-5.6-terra	Balances intelligence and cost.
gpt-5.6-luna	Cost-sensitive, high-volume workloads.
gpt-5.5	Previous-generation frontier model for complex coding, computer use, knowledge work, and research.
gpt-5.4	Frontier model for professional work with strong coding, reasoning, tool use, and agentic workflows.
gpt-5.4-mini	Fast, efficient mini model for responsive coding tasks and subagents.
gpt-5.3-codex-spark	Text-only research preview optimized for near-instant coding iteration; listed for ChatGPT Pro users.

Deprecated for ChatGPT sign-in in Codex: gpt-5.2 and gpt-5.3-codex. Update old scripts/configs that still reference them. Some deprecated ChatGPT-sign-in models may still be available through API-key authentication, so check the API models page if you rely on one.

Sources: Claude Code model configuration, Claude model overview, ChatGPT/Codex models docs, Codex CLI docs, and OpenAI model docs.

If it stops or crashes, run the same command again — it skips rows already in output/results.csv and continues.

If Claude or Codex reports a usage/rate/quota limit, or says acme-crm needs authorization/login, the script saves that row as an error, stops before running the next prompt, and can be resumed later with the same command.

Automatic follow-ups (when you can't pre-write them)

Some prompts come back asking you to confirm, clarify, or choose — and you can't know the right follow-up until you see that response. Add --auto-followup:

python3 run_batch.py --csv prompts.csv --auto-followup

How it works: your prompt runs untouched as turn 1. If the reply asks for input, a separate decider call using the same agent picks the most reasonable proceed-reply (approve confirmations, choose the likeliest option, invent plausible values) and sends it back in the same session — repeating until the task completes or --max-followups (default 3) is hit. If turn 1 is already a complete answer, the decider says DONE and no follow-up runs. --auto-followup ignores the CSV follow-up column.

The full multi-turn exchange is saved in the transcript_json column and the raw log.

Two caveats: (1) it makes extra calls per prompt (turn 1 + a decider check + each follow-up), so it uses more of your subscription quota — pair it with --limit and --delay. (2) The decider makes its own choices, so on a sandbox that's fine, but don't point auto mode at a real environment unless you're happy for it to pick.

A fixed final turn — verify write actions (--final-prompt + Write column)

The CSV follow-up column and the auto-decider both stop as soon as the task looks complete, so neither reliably runs a verification step of its own — and --auto-followup ignores the follow-up column entirely. To always make the model double-check its own writes, the script sends a fixed final turn after any follow-up or auto-followup rounds resolve. It runs in both modes.

Two things make it convenient:

• The text is fixed and built in. It lives in DEFAULT_FINAL_PROMPT at the top of run_batch.py — edit it there; you don't repeat it on the command line. So the final turn is on by default:

  python3 run_batch.py --csv prompts.csv --auto-followup
  

  Pass --final-prompt "..." only to override the text for one run, or --final-prompt "" to turn it off.

• It only fires for write actions. Add a Write column (yes/no) to your CSV; the final turn runs only on rows marked yes, so read-only prompts don't get a pointless "verify the write" turn. Blank cells count as a write; if the Write column is absent entirely, every row is treated as a write (and the script prints a note reminding you).

Because it always runs last, verification happens even when the earlier turns were dynamic (auto mode) or when turn 1 was already a complete answer. The final turn shows up as kind: "final_prompt" in transcript_json, and its answer becomes the followup_response (final) column. Note it adds one extra call per write-row, and (like any turn here) it can fire real write/read tools — point it at a sandbox.

Replacing prompts.csv with new content (same filename)

Resume matches on row position, not prompt text. So if you swap in a new prompts.csv after a partial run, the old rows would be wrongly skipped. Two fixes:

• Start clean but keep the old data (archives the previous run automatically):

  python3 run_batch.py --csv prompts.csv --fresh
  

• Or keep each version fully separate with its own output folder:

  python3 run_batch.py --csv prompts.csv --outdir output_v2
  

Output

• output/results.csv — one row per prompt. Key columns:
  • tools_called — every tool that fired across all turns, e.g. mcp__acme-crm__get_whos_out
  • tool_call_count — total tool calls across all turns
  • tool_calls_json — full tool names + the exact inputs the agent sent (turn 1)
  • response — the agent's turn-1 answer
  • turns — how many turns ran (1 = no follow-up; 2+ = follow-ups happened)
  • followup — the follow-up text sent (the auto-decider's reply, in auto mode)
  • followup_tools_called, followup_response — tools + final answer after follow-ups
  • transcript_json — the full multi-turn exchange (each turn's input, tools, response)
  • cost_usd, num_turns, is_error, error, session_id
• output/raw/row_XXXX.jsonl — the complete event stream per prompt (every tool call, tool result, and message) for deep inspection.

Good to know

• This uses your plan's usage. 100+ prompts, each with tool calls, draw on the selected CLI account's limits — you may hit rate/usage caps on a big batch. Pace with --delay, pilot with --limit, and rely on auto-resume to spread a run across sittings.
• Write actions run for real. Prompts that create/update records (submit, update, terminate, etc.) execute against whatever acme-crm points to. Re-running after deleting results.csv re-fires them.
• Claude is locked to acme-crm + Bash. The Claude adapter passes --allowedTools mcp__acme-crm,Bash and --permission-mode dontAsk.
• Codex uses registered MCP config. If Codex reports an OAuth refresh or authorization error, run codex mcp login acme-crm and retry.