Skip to content

fix(llm): validate the /health body and identify OpenAI-compatible runners - #73

Open
sosidudku1 wants to merge 1 commit into
mainfrom
fix/external-url-probe-honesty
Open

fix(llm): validate the /health body and identify OpenAI-compatible runners#73
sosidudku1 wants to merge 1 commit into
mainfrom
fix/external-url-probe-honesty

Conversation

@sosidudku1

Copy link
Copy Markdown
Collaborator

Closes #65, closes #66.

What actually happened in #65

KoboldCpp serves its web UI with HTTP 200 on every path, including /health. The external-URL probe treated any 200 as success, so it passed falsely, the chat route switched onto a server the llama.cpp client cannot drive, and the next call hung while the busy flag ate the keyboard. That is the permanent freeze the reporter described for http://localhost:5001. With /v1 appended the downstream call failed fast instead, which is why that variant unfroze after seconds. (The earlier theory in the issue thread blamed probe retries; the probe on this path already ran a single 8s attempt, the hang lived after the false pass.)

Change

  1. A 200 on /health is no longer enough. The body must carry llama.cpp's JSON shape (a status string; ok and loading model both count). KoboldCpp's HTML now fails the probe instead of hijacking the route.
  2. A second probe identifies OpenAI-compatible runners. When /health says no, {base}/v1/models is asked; a JSON answer with a data array marks the result kind: "openai-compat" (KoboldCpp, LM Studio, vLLM all match). The external llama.cpp route genuinely cannot drive these servers, so passing them through would only move the hang one step later.
  3. The save path talks. probing <url> is emitted immediately (an 8s silent probe reads as a freeze), and an openai-compat answer produces an actionable message: add the server as a cloud provider, base URL included, which is the path that actually works for these runners today.

Testing

New llama-server-health.test.ts (the module had no tests): real llama /health accepted, loading model accepted, KoboldCpp-style HTML 200 rejected with a clear error, the /v1/models fallback identifying an OpenAI-compatible runner, network failure reporting unknown, and retry short-circuiting on first success.

All probe consumers pass unchanged: src/llm, src/tui/llm-health, src/http, src/runtime — 356 tests. tsc clean. Full suite shows the same 8 pre-existing failures as main.

…nners

Closes #65, closes #66. KoboldCpp answers HTTP 200 with its web UI on
every path, including /health, so the external-URL probe passed falsely,
the chat route switched onto a server the llama.cpp client cannot drive,
and the session hung with the keyboard eaten by the busy flag. With /v1
appended the downstream call failed fast instead, which is why the user
saw a short freeze in one case and a permanent one in the other.

- a 200 on /health now counts only when the body carries llama.cpp's
  JSON shape (a status string); anything else is not a llama-server
- when /health says no, a secondary probe asks {base}/v1/models; a JSON
  answer with a data array identifies an OpenAI-compatible runner
  (KoboldCpp, LM Studio, vLLM) and the result carries kind:
  "openai-compat"
- the URL-save path now emits "probing <url>" immediately so the 8s
  probe window no longer reads as a freeze, and on an openai-compat
  answer it tells the operator exactly how to add the server as a cloud
  provider instead of failing with a bare error

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: Please add support to other LLM runners Lag and instability issues

1 participant