Skip to content

decisiongrounding: OpenAI-compatible (LiteLLM) answering backend — first real-gateway validation #11

Description

@tcballard

What

The decisiongrounding benchmark now supports OpenAI-compatible gateways (the common enterprise LiteLLM config) via a new answering backend:

  • --answering litellm:<model-alias> (OpenAICompatAnsweringModel), speaking /chat/completions + response_format: json_schema over stdlib urllib — no new dependency.
  • Held-constant contract preserved (DG-ADR-0001): the litellm and claude backends share the same scaffold, user-prompt builder, and ProposedChange schema; only the wire format differs. Covered by tests, including a localhost mock-gateway round-trip.
  • scripts/litellm_probe.py --mode openai probes the exact adapter request before any funded spend.
  • ANSWERING=litellm:<alias> ./scripts/run_real.sh wired; --batch/BATCH=1 refuse the combination (no Batch API on the OpenAI surface).
  • Recorded in decisiongrounding/rac/decisions/ADR-0005-openai-compat-answering-adapter.md (Accepted).

Remaining work this issue tracks

  • Validate against a real LiteLLM gateway (both probe modes; confirm response_format json_schema translation for the chosen alias and that usage is forwarded).
  • Confirm the gateway alias is pinned to a fixed model before any recorded run (version honesty rule in ADR-0005).
  • First gateway-routed real run (Step 2 cheap validation from docs/AGENTIC_BENCHMARK_RUN_HANDOFF.md) and note any gateway-specific behaviour back into the README proxy section.

Why

Enterprise adopters route all model traffic through LiteLLM; supporting only the Anthropic-native passthrough made the benchmark unrunnable exactly where the rac-grounding thesis gets evaluated for adoption.

Roadmap: decisiongrounding/rac/roadmaps/publish-swe-decisionbench.md (multi-model generalization initiative — per ADR-093, execution tracked here).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions