fix(llm): validate the /health body and identify OpenAI-compatible runners - #73
Open
sosidudku1 wants to merge 1 commit into
Open
fix(llm): validate the /health body and identify OpenAI-compatible runners#73sosidudku1 wants to merge 1 commit into
sosidudku1 wants to merge 1 commit into
Conversation
…nners Closes #65, closes #66. KoboldCpp answers HTTP 200 with its web UI on every path, including /health, so the external-URL probe passed falsely, the chat route switched onto a server the llama.cpp client cannot drive, and the session hung with the keyboard eaten by the busy flag. With /v1 appended the downstream call failed fast instead, which is why the user saw a short freeze in one case and a permanent one in the other. - a 200 on /health now counts only when the body carries llama.cpp's JSON shape (a status string); anything else is not a llama-server - when /health says no, a secondary probe asks {base}/v1/models; a JSON answer with a data array identifies an OpenAI-compatible runner (KoboldCpp, LM Studio, vLLM) and the result carries kind: "openai-compat" - the URL-save path now emits "probing <url>" immediately so the 8s probe window no longer reads as a freeze, and on an openai-compat answer it tells the operator exactly how to add the server as a cloud provider instead of failing with a bare error Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #65, closes #66.
What actually happened in #65
KoboldCpp serves its web UI with HTTP 200 on every path, including
/health. The external-URL probe treated any 200 as success, so it passed falsely, the chat route switched onto a server the llama.cpp client cannot drive, and the next call hung while the busy flag ate the keyboard. That is the permanent freeze the reporter described forhttp://localhost:5001. With/v1appended the downstream call failed fast instead, which is why that variant unfroze after seconds. (The earlier theory in the issue thread blamed probe retries; the probe on this path already ran a single 8s attempt, the hang lived after the false pass.)Change
/healthis no longer enough. The body must carry llama.cpp's JSON shape (astatusstring;okandloading modelboth count). KoboldCpp's HTML now fails the probe instead of hijacking the route./healthsays no,{base}/v1/modelsis asked; a JSON answer with adataarray marks the resultkind: "openai-compat"(KoboldCpp, LM Studio, vLLM all match). The external llama.cpp route genuinely cannot drive these servers, so passing them through would only move the hang one step later.probing <url>is emitted immediately (an 8s silent probe reads as a freeze), and an openai-compat answer produces an actionable message: add the server as a cloud provider, base URL included, which is the path that actually works for these runners today.Testing
New
llama-server-health.test.ts(the module had no tests): real llama /health accepted,loading modelaccepted, KoboldCpp-style HTML 200 rejected with a clear error, the/v1/modelsfallback identifying an OpenAI-compatible runner, network failure reportingunknown, and retry short-circuiting on first success.All probe consumers pass unchanged:
src/llm,src/tui/llm-health,src/http,src/runtime— 356 tests.tscclean. Full suite shows the same 8 pre-existing failures as main.