fix: what the second end-to-end run found - #207
Merged
Conversation
Found by re-running the end-to-end path after #201-#205. Asked "How do I renew a TLS certificate before it expires?", deepseek-v4-flash ran `kb_search` six times — three of them the identical query — hit CHAT_MAX_TURNS, and the user got 62 characters: Let me check the knowledge base for any relevant procedures. with seven citations attached to it and 578 output tokens billed. The loop's exhaustion branch returned the last round's `resp.content`. But the last round produced a *tool call*, so that content is the preamble the model writes before reaching for a tool — never an answer. The comment said "answer with whatever the last turn produced"; the last turn produced a tool call. The cap now bounds the *tool* rounds. When they run out, one more round goes out with no tools: the results are already in `provider_msgs`, and with nothing left to call, the only move is to answer from them. Measured on the same question, same forced condition, live models: before 73 chars "Let me search the knowledge base for TLS certificate…" after 1561 chars the certbot / DigiCert renewal procedure, 5 citations The final prompt is *rebuilt*, not extended. `system_prompt` tells the model to call kb_search before answering, to call report_conflict *before it answers*, and which skills it may load — three instructions it can no longer follow, and the conflict one is a precondition it would be stuck on. PROPOSAL_HINT survives, because offering a fact to Memory is prose rather than a tool call. The instruction rides the system prompt rather than an appended user turn: `role="tool"` renders as a `tool_result` block inside a *user* message on Anthropic, so appending one more user message would stack two in a row. Verified against live claude-haiku-4-5 and deepseek-v4-flash — both return a full grounded answer through the exhaustion path. The model repeating a query it has already run is a separate, cheaper problem. It no longer costs the user an answer. behaviour-gate: 6 passed — memory injection 3/3, conflict reported 3/3, distillation keeps dead ends 3/3, proposals stay read-only 3/3, memory proposal 3/3, memory proposal restraint 3/3 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
README.md:214 tells a new user to run
opspilot ingest examples/sample_data_en/kb/
On the repo's own sample data that reported `10 succeeded · 5 failed`. The five
failures were `chunks.jsonl` files raising `AdapterError: unsupported file type`,
and worse, the five `doc-meta.json` sidecars were ingested *as knowledge
documents* — 5 of the 17 chunks in a fresh KB were JSON metadata. Two of them
came back in the top five for "why would a pod be stuck in CrashLoopBackOff".
Neither half is a bug in isolation. That directory was built in 545ae7e as the
sample input for `opspilot kb load-dir`, which recursively loads doc-meta.json +
chunks.jsonl pairs. The README pointed a second, different command at the same
tree later. One directory, two commands, both correct on their own, never run
against each other.
Split it: source documents stay in `kb/`, their frozen projections move to
`fixtures/`. No product code changes, and no README changes either — the command
it already documents is now the one that works.
ingest examples/sample_data_en/kb/ 5 succeeded · 0 failed · 12 chunks
kb load-dir examples/sample_data_en/fixtures/ 5 pairs, ids chk_f3a40001…
The same search that used to surface metadata at ranks 3 and 5 now returns SOP
prose in every position.
`fixtures/README.md` says why they live apart, because the obvious tidy-up is to
move them back.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
**The open decision is closed.** #206 left one deliberately: whether `kb/retrieval.py` belongs on the behaviour gate's protected paths. It does. #203 shipped without gate evidence, and what it changed was which chunks reach the model at all — the input every one of the five prompt-driven behaviours is judged on. The list's own rule already settled it: over-triggering costs minutes, missing the change costs the reason the gate exists. The entry carries that reasoning, because retrieval.py holds no prompt and the next reader will ask why it is there. **Two counts were stale.** #205 added the fifth behaviour and neither the CI comment nor the Makefile banner followed; CI also printed a paste-me example reading `behaviour-gate: 4/4 passed (votes 3/3, 3/3, 3/3, 3/3)`. The gate only greps for the `^behaviour-gate:` prefix, so a contributor copying that example would have landed a permanent "4/4" for a six-case run — in the one artifact the comment above it calls the whole point. **ROADMAP described two shipped things as open.** #175's silent model swap was fixed in #177 (`model_fallback` trace event, result re-labelled), and the proposed-actions preview/execute UI shipped in #190. Both still read as outstanding work. The real gap in proposed actions is elsewhere and now says so: nothing in `playbooks/` opts in, so an escalated Session returns `{"actions": []}` on every fresh install, and outside ROADMAP the key is named nowhere — not in a playbook, not in ADR-0028, which says only that playbooks opt in. **And the second run is recorded.** Ten checks against the nine fixes from #201-#205, ten passes. Three further defects, two fixed in this PR, plus one rough edge: the CLI writes as `cli:<osuser>` while the loopback API writes as `local-dev`, so `opspilot workingset status` reports nothing open while the web UI has a set open for the same person. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vicenteliu
added a commit
that referenced
this pull request
Sep 1, 2026
…dge (#217) /kb list showed 11 docs, five titled "doc-meta" — kb load-dir's sidecar files (doc-meta.json / chunks.jsonl) ingested as documents by the generic directory walk (#214). #207 fixed the sample_data_en case by splitting the directories but left the walk trusting layout; the scn_* examples still pair sidecars with source documents in one directory, so the same one command re-poisons a KB today. discover_files now skips the two reserved sidecar names the way it skips dotfiles: they describe the corpus, they aren't part of it. docs/assets/tui.gif is regenerated against a cleaned-up KB (the five leftover doc-meta documents hard-deleted via `opspilot kb delete`, audit-logged) — the /kb list frame now shows 6 real documents. Closes #214. Claude-Session: https://claude.ai/code/session_01KXrF6HMB12V1hC9xFtooDE Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nine fixes from #201–#205 all changed behaviour on one path, and none of
them had been exercised since. #206 said the way to check them is the way they
were found: use it again. So it was used again — fresh
OPSPILOT_HOME, theREADME's own commands, hosted models.
Ten checks, ten passes. Three new defects, two fixed here.
Verified fixed (#201–#205)
init→ingest→servestarts clean withOPENAI_API_KEYset../paths.staging/corpusand../corpuseach1 succeeded · 0 failed1/2 2/1 3/4 4/3— the FTS arm carries a chunk the vector side ranked 7th$0.011577; Sonnet 5 20860/1974 →$0.09219. Both exact against the table$0.000353808fromusage.costworking_set_iddistil --modelanthropic/claude-sonnet-5crosses providers and draftsAlso confirmed along the way: an injected Memory constraint changed an answer,
report_conflictopenedmcf_470af872against a KB chunk, the distilled Skillkept its dead ends and its blank stopping condition, and escalation wrote both
back-references.
Fixed here
An exhausted tool loop echoed its own preamble. Six
kb_searchrounds, 578output tokens billed, 62 characters delivered — "Let me check the knowledge base
for any relevant procedures." — with seven citations attached. The cap now
bounds the tool rounds; one final round with no tools answers from what was
already retrieved. Live, same question, same forced condition: 73 chars → 1561
chars.
One sample directory fed two commands.
examples/sample_data_en/kb/wasbuilt for
kb load-dir; the README later pointedingestat it. Result on therepo's own data:
10 succeeded · 5 failed, plus fivedoc-meta.jsonsidecarsingested as documents, two of which ranked top-five for a real question. Split
into
kb/(sources) andfixtures/(frozen pairs). No product code, no READMEchange — the documented command simply works now.
Also here
kb/retrieval.pyjoins the behaviour gate's protected paths, closing thedecision #206 left open. Two stale behaviour counts corrected, including a
paste-me example in CI that would have recorded "4/4" for a six-case run.
ROADMAP stops describing #175 and the proposed-actions UI as open, and records
the run.
Left open, deliberately
propose_actionsis opt-in and nothing opts in. An escalated Sessionreturns
{"actions": []}on every fresh install. Whether the default should bedark is a decision, not a papercut.
cli:<osuser>vslocal-dev, soopspilot workingset statusreports nothing open while the webUI has a set open for the same person.
distilescapes it by taking anexplicit id;
statusandclosedo not.Verification
Full suite 1351 passed.
pytest -m "not slow and not requires_ollama"— CI'sselection — 1349 passed. ruff, ruff format, mypy(154) clean. The one local
failure is
test_providers_ollama.py::TestIntegration::test_chat_smoke, whichwants
gemma4:e4bfrom a local Ollama that does not have it; CI deselects it.behaviour-gate: 6 passed — memory injection 3/3, conflict reported 3/3, distillation keeps dead ends 3/3, proposals stay read-only 3/3, memory proposal 3/3, memory proposal restraint 3/3
Run twice, because
chat_agent.pychanged twice. The memory-proposal case scored3/3 both times, against 2/3 when #205 landed — one session cannot tell an
improvement from sampling noise, so the votes are reported rather than claimed.
🤖 Generated with Claude Code