Skip to content

fix: what the second end-to-end run found - #207

Merged
vicenteliu merged 3 commits into
mainfrom
fix/the-second-end-to-end-run
Aug 19, 2026
Merged

fix: what the second end-to-end run found#207
vicenteliu merged 3 commits into
mainfrom
fix/the-second-end-to-end-run

Conversation

@vicenteliu

Copy link
Copy Markdown
Owner

The nine fixes from #201#205 all changed behaviour on one path, and none of
them had been exercised since. #206 said the way to check them is the way they
were found: use it again. So it was used again — fresh OPSPILOT_HOME, the
README's own commands, hosted models.

Ten checks, ten passes. Three new defects, two fixed here.

Verified fixed (#201#205)

check evidence
embedders agree initingestserve starts clean with OPENAI_API_KEY set
dot-dir / ../ paths .staging/corpus and ../corpus each 1 succeeded · 0 failed
hybrid on a sentence ranks 1/2 2/1 3/4 4/3 — the FTS arm carries a chunk the vector side ranked 7th
Anthropic cost Haiku 8022/711 → $0.011577; Sonnet 5 20860/1974 → $0.09219. Both exact against the table
OpenAI-compatible cost OpenRouter → $0.000353808 from usage.cost
Memory proposal hint fires on both turns, and the second proposes something the first did not
working_set_id present in the list and the detail response
bundle round-trip imported into an empty home; 2 entries restored with their original actors
distil --model anthropic/claude-sonnet-5 crosses providers and drafts
model-list editor no-op rewrite byte-identical; adding a model keeps 21/21 comments

Also confirmed along the way: an injected Memory constraint changed an answer,
report_conflict opened mcf_470af872 against a KB chunk, the distilled Skill
kept its dead ends and its blank stopping condition, and escalation wrote both
back-references.

Fixed here

An exhausted tool loop echoed its own preamble. Six kb_search rounds, 578
output tokens billed, 62 characters delivered — "Let me check the knowledge base
for any relevant procedures." — with seven citations attached. The cap now
bounds the tool rounds; one final round with no tools answers from what was
already retrieved. Live, same question, same forced condition: 73 chars → 1561
chars.

One sample directory fed two commands. examples/sample_data_en/kb/ was
built for kb load-dir; the README later pointed ingest at it. Result on the
repo's own data: 10 succeeded · 5 failed, plus five doc-meta.json sidecars
ingested as documents, two of which ranked top-five for a real question. Split
into kb/ (sources) and fixtures/ (frozen pairs). No product code, no README
change — the documented command simply works now.

Also here

kb/retrieval.py joins the behaviour gate's protected paths, closing the
decision #206 left open. Two stale behaviour counts corrected, including a
paste-me example in CI that would have recorded "4/4" for a six-case run.
ROADMAP stops describing #175 and the proposed-actions UI as open, and records
the run.

Left open, deliberately

  • propose_actions is opt-in and nothing opts in. An escalated Session
    returns {"actions": []} on every fresh install. Whether the default should be
    dark is a decision, not a papercut.
  • CLI and API disagree on who the local operator iscli:<osuser> vs
    local-dev, so opspilot workingset status reports nothing open while the web
    UI has a set open for the same person. distil escapes it by taking an
    explicit id; status and close do not.

Verification

Full suite 1351 passed. pytest -m "not slow and not requires_ollama" — CI's
selection — 1349 passed. ruff, ruff format, mypy(154) clean. The one local
failure is test_providers_ollama.py::TestIntegration::test_chat_smoke, which
wants gemma4:e4b from a local Ollama that does not have it; CI deselects it.

behaviour-gate: 6 passed — memory injection 3/3, conflict reported 3/3, distillation keeps dead ends 3/3, proposals stay read-only 3/3, memory proposal 3/3, memory proposal restraint 3/3

Run twice, because chat_agent.py changed twice. The memory-proposal case scored
3/3 both times, against 2/3 when #205 landed — one session cannot tell an
improvement from sampling noise, so the votes are reported rather than claimed.

🤖 Generated with Claude Code

vicenteliu and others added 3 commits August 19, 2026 05:19
Found by re-running the end-to-end path after #201-#205. Asked "How do I renew
a TLS certificate before it expires?", deepseek-v4-flash ran `kb_search` six
times — three of them the identical query — hit CHAT_MAX_TURNS, and the user
got 62 characters:

    Let me check the knowledge base for any relevant procedures.

with seven citations attached to it and 578 output tokens billed.

The loop's exhaustion branch returned the last round's `resp.content`. But the
last round produced a *tool call*, so that content is the preamble the model
writes before reaching for a tool — never an answer. The comment said "answer
with whatever the last turn produced"; the last turn produced a tool call.

The cap now bounds the *tool* rounds. When they run out, one more round goes out
with no tools: the results are already in `provider_msgs`, and with nothing left
to call, the only move is to answer from them. Measured on the same question,
same forced condition, live models:

    before   73 chars   "Let me search the knowledge base for TLS certificate…"
    after  1561 chars   the certbot / DigiCert renewal procedure, 5 citations

The final prompt is *rebuilt*, not extended. `system_prompt` tells the model to
call kb_search before answering, to call report_conflict *before it answers*,
and which skills it may load — three instructions it can no longer follow, and
the conflict one is a precondition it would be stuck on. PROPOSAL_HINT survives,
because offering a fact to Memory is prose rather than a tool call.

The instruction rides the system prompt rather than an appended user turn:
`role="tool"` renders as a `tool_result` block inside a *user* message on
Anthropic, so appending one more user message would stack two in a row. Verified
against live claude-haiku-4-5 and deepseek-v4-flash — both return a full
grounded answer through the exhaustion path.

The model repeating a query it has already run is a separate, cheaper problem.
It no longer costs the user an answer.

behaviour-gate: 6 passed — memory injection 3/3, conflict reported 3/3,
distillation keeps dead ends 3/3, proposals stay read-only 3/3, memory proposal
3/3, memory proposal restraint 3/3

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
README.md:214 tells a new user to run

    opspilot ingest examples/sample_data_en/kb/

On the repo's own sample data that reported `10 succeeded · 5 failed`. The five
failures were `chunks.jsonl` files raising `AdapterError: unsupported file type`,
and worse, the five `doc-meta.json` sidecars were ingested *as knowledge
documents* — 5 of the 17 chunks in a fresh KB were JSON metadata. Two of them
came back in the top five for "why would a pod be stuck in CrashLoopBackOff".

Neither half is a bug in isolation. That directory was built in 545ae7e as the
sample input for `opspilot kb load-dir`, which recursively loads doc-meta.json +
chunks.jsonl pairs. The README pointed a second, different command at the same
tree later. One directory, two commands, both correct on their own, never run
against each other.

Split it: source documents stay in `kb/`, their frozen projections move to
`fixtures/`. No product code changes, and no README changes either — the command
it already documents is now the one that works.

    ingest examples/sample_data_en/kb/         5 succeeded · 0 failed · 12 chunks
    kb load-dir examples/sample_data_en/fixtures/   5 pairs, ids chk_f3a40001…

The same search that used to surface metadata at ranks 3 and 5 now returns SOP
prose in every position.

`fixtures/README.md` says why they live apart, because the obvious tidy-up is to
move them back.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
**The open decision is closed.** #206 left one deliberately: whether
`kb/retrieval.py` belongs on the behaviour gate's protected paths. It does. #203
shipped without gate evidence, and what it changed was which chunks reach the
model at all — the input every one of the five prompt-driven behaviours is
judged on. The list's own rule already settled it: over-triggering costs
minutes, missing the change costs the reason the gate exists. The entry carries
that reasoning, because retrieval.py holds no prompt and the next reader will
ask why it is there.

**Two counts were stale.** #205 added the fifth behaviour and neither the CI
comment nor the Makefile banner followed; CI also printed a paste-me example
reading `behaviour-gate: 4/4 passed (votes 3/3, 3/3, 3/3, 3/3)`. The gate only
greps for the `^behaviour-gate:` prefix, so a contributor copying that example
would have landed a permanent "4/4" for a six-case run — in the one artifact the
comment above it calls the whole point.

**ROADMAP described two shipped things as open.** #175's silent model swap was
fixed in #177 (`model_fallback` trace event, result re-labelled), and the
proposed-actions preview/execute UI shipped in #190. Both still read as
outstanding work. The real gap in proposed actions is elsewhere and now says so:
nothing in `playbooks/` opts in, so an escalated Session returns `{"actions":
[]}` on every fresh install, and outside ROADMAP the key is named nowhere — not
in a playbook, not in ADR-0028, which says only that playbooks opt in.

**And the second run is recorded.** Ten checks against the nine fixes from
#201-#205, ten passes. Three further defects, two fixed in this PR, plus one
rough edge: the CLI writes as `cli:<osuser>` while the loopback API writes as
`local-dev`, so `opspilot workingset status` reports nothing open while the web
UI has a set open for the same person.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@vicenteliu
vicenteliu merged commit 0706a7b into main Aug 19, 2026
4 checks passed
@vicenteliu
vicenteliu deleted the fix/the-second-end-to-end-run branch August 19, 2026 12:22
vicenteliu added a commit that referenced this pull request Sep 1, 2026
…dge (#217)

/kb list showed 11 docs, five titled "doc-meta" — kb load-dir's sidecar
files (doc-meta.json / chunks.jsonl) ingested as documents by the
generic directory walk (#214). #207 fixed the sample_data_en case by
splitting the directories but left the walk trusting layout; the scn_*
examples still pair sidecars with source documents in one directory,
so the same one command re-poisons a KB today.

discover_files now skips the two reserved sidecar names the way it
skips dotfiles: they describe the corpus, they aren't part of it.

docs/assets/tui.gif is regenerated against a cleaned-up KB (the five
leftover doc-meta documents hard-deleted via `opspilot kb delete`,
audit-logged) — the /kb list frame now shows 6 real documents.

Closes #214.


Claude-Session: https://claude.ai/code/session_01KXrF6HMB12V1hC9xFtooDE

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant