LLM API Doctor

Zero-dependency conformance checks for OpenAI-compatible Chat Completions endpoints.

It answers a narrow question: will an ordinary OpenAI-compatible client actually understand this endpoint?

The doctor checks model discovery, a non-streaming completion, SSE framing, HTTP 200 error objects, missing choices, non-standard success/data wrappers, and an optional forced tool call. It reports structure only—never generated text or API-key values.

Quick start

Requirements: Node.js 22 or newer.

git clone https://github.com/ken050210/llm-api-doctor.git
Set-Location .\llm-api-doctor

$env:OPENAI_API_KEY = "your-key"
node .\bin\llm-api-doctor.mjs `
  --base-url https://api.example.com/v1 `
  --model your-model

Run every optional probe and emit a CI-friendly JSON report:

node .\bin\llm-api-doctor.mjs `
  --base-url https://api.example.com/v1 `
  --model your-model `
  --with-tools `
  --format json

If --model is omitted, the doctor uses OPENAI_MODEL or the first model returned by GET /models.

Checks

Check	Default	What fails it
GET /models	on	HTTP error, invalid JSON, missing/empty data, requested model absent
Non-stream completion	on	HTTP error, invalid JSON, error object, wrapped response, missing choices/message
SSE streaming	on	wrong content type, invalid JSON event, missing choices, missing [DONE]
Forced tool call	off	missing named tool call or non-JSON arguments

The normal completion also reports whether the response contains usage, tool calls, or common reasoning fields. Those fields are informational because reasoning controls are provider-specific.

Options

--base-url URL
--model ID
--api-key-env NAME        default: OPENAI_API_KEY
--format text|json|markdown
--timeout-ms NUMBER       default: 60000
--skip-stream
--with-tools
--allow-insecure-http

Use --help for the complete command reference. Exit code 0 means every enabled check passed, 1 means at least one compatibility check failed, and 2 means the command itself was invalid.

Security and privacy

• API keys are accepted only through an environment variable or the programmatic API.
• Reports never include the key or generated response content.
• Redirects are refused, so an authorization header is not forwarded to another origin.
• Response bodies are bounded to prevent untrusted endpoints from exhausting memory.
• Plaintext HTTP is accepted by default only for localhost, 127.0.0.1, and ::1.

Development

npm test
npm run verify

Tests use local mock servers and make no provider requests.

中文说明

这是一个零依赖的 OpenAI-compatible API 兼容性体检工具。它会检查 /models、普通 Chat Completions、SSE、HTTP 200 错误对象、success/data 包装响应，以及可选的强制工具调用。

密钥只从环境变量读取；报告不会包含密钥、提示词或模型生成内容。默认只允许回环地址使用明文 HTTP。

License

MIT