OpenAI-compat: --model openai-chat-completions with full chat URL, not curator or vLLM

[cursor]

Working config (--model openai-chat-completions + full chat URL)

Evalchemy's documented OpenAI path is --model openai-chat-completions. That class posts base_url as-is (default https://api.openai.com/v1/chat/completions). A /v1 root 404s. --model curator is LiteLLM. --model vllm is local serving.

A Chat Completions host whose public catalog is GET https://api.pzero.studio/v1/models (no key) fits if model_args includes a catalog model and base_url is the full chat URL.

export OPENAI_API_KEY="<Bearer key for that host>"

python -m eval.eval \
  --model openai-chat-completions \
  --tasks alpaca_eval \
  --model_args "model=deepseek-v4-flash,base_url=https://api.pzero.studio/v1/chat/completions,num_concurrent=1" \
  --output_path logs

OPENAI_API_KEY is required. Do not pass --model_name (that injects HuggingFace pretrained=).