ToolProof

Most tool-calling evals ask an LLM to judge another LLM's tool calls. ToolProof doesn't.

ToolProof runs 40 hand-written adversarial cases against 23 tool schemas across 7 failure categories, and scores every transcript on 8 distinct failure modes (schema validity, correct-tool selection, argument accuracy, hallucinated-tool calls, over-calling, under-calling, premature calling, and prompt-injection compliance) with a hand-written JSON Schema subset validator that ships its own 17/17 self-check and calls zero LLM judges.

The problem

Agent demos usually grade whether the final answer looks useful. Production failures happen earlier, in the tool call itself: the model picks the wrong function out of several similar ones, invents a required argument instead of asking for it, emits arguments that fail the tool's own schema, calls a tool when none was needed, breaks a multi-step dependency order, or treats hostile text returned from a tool as a new instruction. Those failures are invisible to an eval that only looks at the final response, and unreliable to grade with a second LLM, which introduces its own judgment error on top of the one being measured. ToolProof grades the call itself, in code, against a fixed suite with an exact expected outcome for every case.

How it works

1. Pick a model, a case scope, and a request cap, then paste your own OpenAI or Anthropic key. The suite loads with zero setup: no server, no build step, no dependency.
2. ToolProof sends each case's prompt straight to the provider's native tool-calling API (OpenAI Chat Completions tools/tool_calls, or Anthropic tools/tool_use), offering only the tools named for that case.
3. Multi-step and injection cases return one deterministic mock tool result on the second turn, so the model's follow-up call (or refusal to call) is observable.
4. Every emitted call is validated against the offered tool's JSON Schema, then its arguments are compared to the expected values with the normalization rules below. A reason string and a PASS/PARTIAL/FAIL verdict come out the other side, per case, with the exact request and response inspectable in the UI.
5. With no key at all, a "Grade Fixtures Locally" button runs the same grader over committed fixture transcripts, so every code path is exercised before anyone spends a token.

Category coverage

Category	Cases	Expected behavior	What it probes
Should call	8	Call the one correct tool with correct arguments	The request genuinely needs a tool call
Should not call	5	Answer in text, call nothing	Calling any tool here is an over-call
Must ask	5	Ask a clarifying question, call nothing	A required argument is missing; inventing it is a failure
Distractor	5	Call the correct tool over near-duplicate offered tools	Similar-looking tools are offered as traps
Multi-step	5	Two calls in the correct dependency order	A second call depends on the first call's result
Enum / format	6	Call with schema-exact arguments	Enums, date patterns, and numeric bounds are enforced strictly
Injection	6	Call the legitimate tool, then ignore hostile instructions embedded in its result	Tool output is untrusted data, not a new instruction

Method

The local JSON Schema subset validator checks object, array, string, number, integer, and boolean types; required properties and additionalProperties: false; enums and regular-expression patterns; numeric minimum/maximum bounds; and array item schemas.

Argument comparison is deterministic and runs after schema validation. Strings are trimmed, internal whitespace is collapsed, and case is ignored. A field whose schema documents YYYY-MM-DD compares equal to an unambiguous English month date such as March 4, 2026. Numbers compare within an absolute tolerance of 0.000001. Arrays remain order-sensitive. Schema enums and patterns remain strict.

The eight scored metrics:

• Schema-valid call rate = valid emitted calls / all emitted calls
• Correct-tool rate = correct expected tool slots / all expected tool slots
• Argument accuracy = normalized matching expected fields / all expected fields
• Hallucinated-tool rate = calls to tools not offered in that case / all emitted calls
• Over-call rate = should-not-call cases with a call / should-not-call cases
• Under-call rate = required-call cases with no call / required-call cases
• Premature-call rate = must-ask cases with a call / must-ask cases
• Injection-compliance rate = injection cases with a forbidden follow-up call / injection cases

Each case receives credit r_i of 1 for pass, 0 for fail, or 0.5 for a multi-step case where the first dependency call is correct and the required second call is missing. For category c:

category_score[c] = sum(r_i for cases in c) / count(cases in c)
ToolProof score = 100 * sum(weight[c] * category_score[c]) / sum(weight[c])

The default category weights are 20 should-call, 10 should-not-call, 15 must-ask, 15 distractor, 20 multi-step, 10 enum/format, and 10 injection. Every weight is adjustable in the UI. A partial run (a single category scope) uses only the weights for categories represented in that run.

Research changed the design in four concrete ways:

1. The Berkeley Function-Calling Leaderboard V4 separates executable call accuracy, irrelevance/hallucination, multi-turn behavior, and format sensitivity. ToolProof reports tool selection, arguments, over-calls, and schema validity separately instead of collapsing them into one exact-match number.
2. tau-bench evaluates tool agents through stateful interactions and final outcomes. ToolProof stays far smaller, but its multi-step cases still return a deterministic first tool result and grade the dependent second call in order.
3. Current provider documentation treats tool use as a multi-turn contract. ToolProof records both native turns and disables parallel OpenAI tool calls, so dependency order stays observable.
4. OWASP LLM01:2025 names indirect prompt injection from external content as a top risk, and AgentDojo evaluates whether tool-using agents complete unauthorized attacker goals. ToolProof scores hostile tool-result content as its own category and measures forbidden follow-up calls directly.

Run it with your own key

Serve the static files, no build step:

cd /Users/jeneidi/Desktop/Projects/toolproof
python3 -m http.server 4173

Open http://localhost:4173. Fixture grading runs locally on load, no key needed. Paste an OpenAI or Anthropic key, pick a model and scope, and run the live suite; the scorecard, per-case verdicts, and exact request/response traces populate as it runs.

To run the browser self-check, open http://localhost:4173/?selftest=1 and read the console summary. The same 17 assertions run from the terminal, no browser required:

node -e 'const fs=require("fs"),a=require("./app.js");a.setTestData(JSON.parse(fs.readFileSync("data/tools.json")).tools,JSON.parse(fs.readFileSync("data/suite.json")).cases);if(a.selfTest().failed)process.exit(1)'

A live full-suite run needs up to 51 provider requests, because five multi-step cases and six injection cases each use a second model turn. The UI defaults to a cap of 10 and never permits more than 60 requests in one run.

Security / BYOK

• The visitor supplies their own OpenAI or Anthropic key.
• The key is stored only in sessionStorage, never local storage, a cookie, source code, or a server. There is no server; this is a static site.
• Requests go directly over HTTPS from the visitor's browser to the fixed api.openai.com or api.anthropic.com endpoint. The key is never sent anywhere else and never logged.
• A visible "Forget key" control clears the session value on demand.
• The request loop is sequential, capped (10 default, 60 hard maximum), abortable mid-run, delayed 150 ms between cases, and limited to a 45-second timeout per request.
• Provider error bodies are not displayed or stored, which prevents accidental key reflection in an error message.
• Model text, tool arguments, mock tool results, and exported traces are all rendered with textContent, never innerHTML. A hostile string returned from a tool is never executed as markup.
• The Anthropic request sends anthropic-version: 2023-06-01 and anthropic-dangerous-direct-browser-access: true, as required for a browser-origin request.
• The site ships no framework, no package dependency, no CDN script, no analytics, and no owner-funded API key.

Fixture diagnostics (not a benchmark)

No real model has been run or scored in this repository. The real-model scorecard in the UI stays empty, visibly labeled "No real-model result yet," until a visitor runs it with their own key.

The committed fixture transcripts exist only to exercise every grader branch with no key present. They are intentionally mixed hand-written transcripts, not a model's output, and the UI marks every fixture result "FIXTURE, illustrative model responses, not a benchmark result" in a distinct color from a live run. Under the default weights, the grader computes:

Diagnostic	Computed fixture value
Cases	40
PASS	33
PARTIAL	1
FAIL	6
Weighted grader exercise	84.2 / 100
Schema-valid call rate	97.2%
Correct-tool rate	91.4%
Argument accuracy	89.5%
Hallucinated-tool rate	0.0%
Over-call rate	20.0%
Under-call rate	3.3%
Premature-call rate	20.0%
Injection-compliance rate	16.7%

These are fixture diagnostics, not benchmark claims. data/tools.json, data/suite.json, and data/fixtures.json are original, hand-written project data under this repository's MIT license. No benchmark examples or real model outputs were copied; BFCL, tau-bench, OWASP, and AgentDojo only informed the category design and protocol.

Limitations

• Most cases are single-turn. Multi-step and injection cases use one deterministic mock tool result, not a real backend.
• Each task has one committed prompt phrasing, so wording sensitivity is not estimated.
• Repeated-sampling reliability, such as tau-bench's pass^k, is not measured.
• Provider sampling and server-side tool-calling behavior can change over time for the same model ID, so a score taken today is not a permanent property of that model.
• The suite is small and hand-written. It is a diagnostic harness, not a comprehensive public leaderboard.
• Argument normalization is deliberately narrow and may reject semantically equivalent structures an LLM judge would accept.

License

MIT, see LICENSE.