Skip to content

#480 fix: silently strip from_* tool calls before dispatch#487

Merged
bonk-moltbot merged 2 commits into
mainfrom
fix/480-strip-phantom-tool-calls
Apr 30, 2026
Merged

#480 fix: silently strip from_* tool calls before dispatch#487
bonk-moltbot merged 2 commits into
mainfrom
fix/480-strip-phantom-tool-calls

Conversation

@hoblin

@hoblin hoblin commented Apr 30, 2026

Copy link
Copy Markdown
Owner

Summary

Implements the fix for #480 based on the research-spike findings recorded in the comment on the issue.

The spike (PR #485) confirmed that spurious from_* tool calls — from_shell-runner, from_zero-width-sleuth, from_melete_goal, etc. — are LLM hallucinations, not internal leaks. The model emits them as legitimate tool_use blocks in the raw Anthropic response, complete with fabricated input schemas, despite the from_* naming convention from PR #445 + system-prompt sister block + a Level 2 memory explicitly documenting the anti-pattern. Three reproductions in ~3 minutes of a single deliberately-constructed session.

What this PR does

A small pure filter — Aoide::PhantomCallFilter.call(hash) → hash — drops every block where type == "tool_use" && name.start_with?("from_") from response["content"] before the rest of the main loop sees it. The filter has no AR, no events, no I/O — pure hash transformation, easy to spec in isolation.

It runs at the top of Events::Subscribers::LLMResponseHandler#emit, immediately after log_raw_response so the dev-only diagnostic log from PR #485 keeps observing the pre-filter API response. This matters: we want to keep measuring hallucination rate after the fix lands, and decide later whether a feedback loop (synthesised tool_result, Mneme nudge) is worth adding.

After the strip, the existing handler logic carries the rest of the work unchanged:

  • normalize_tool_uses operates on the filtered content → no from_* enters dispatch.
  • extract_text is unaffected by the strip → text always survives.
  • The if tool_uses.any? … elsif session.may_response_complete? … branch handles the empty case naturally → from_*-only responses land at clean response_complete!.

Three edge cases the user explicitly asked to cover

Response shape Behaviour
Only from_* tool calls No dispatch, no failed tool_result, session → idle (clean end of turn)
from_* + legitimate tool calls Only legitimate calls dispatch; session → executing; one tool_call message persisted
Text + from_* tool calls Text persisted as agent_message; no dispatch; session → idle

All three are asserted in spec/lib/events/subscribers/llm_response_handler_spec.rb. The filter itself has 13 unit specs against fixture payloads (including symbol-keyed content, missing content key, non-array content, malformed blocks).

Diagnostic continuity

log/aoide.log keeps logging the raw API response and raw tool_use blocks pre-filter. Post-fix, dispatching tool=from_* lines should disappear, but from_* blocks may still appear in raw tool_use blocks: — that's the signal that the model is still hallucinating. If frequency stays high we'll know without re-instrumenting.

Test plan

  • bundle exec rspec spec/lib/aoide/phantom_call_filter_spec.rb — 13 examples, 0 failures
  • bundle exec rspec spec/lib/events/subscribers/llm_response_handler_spec.rb — 16 examples (3 new edge cases), 0 failures
  • bundle exec rspec spec/jobs/drain_job_spec.rb — 11 examples, 0 failures (sanity on upstream emitter)
  • bundle exec standardrb — clean on changed files
  • bundle exec reek lib/aoide/phantom_call_filter.rb lib/events/subscribers/llm_response_handler.rb — 2 pre-existing FeatureEnvy warnings remain, none introduced
  • Smoke test on dev brain — restart on this branch, reproduce the parallel-subagent scenario, confirm dispatching tool=from_* lines no longer appear and no UnknownToolError lands in the conversation. (Doing right after this PR opens.)

Closes #480

hoblin added 2 commits April 30, 2026 11:11
Issue #480's research spike (PR #485) confirmed the spurious from_*
tool calls — `from_shell-runner`, `from_zero-width-sleuth`, etc., that
produce `Tools::UnknownToolError` — are LLM hallucinations, not
internal leaks. The model emits them as real tool_use blocks in the
Anthropic response, complete with a fabricated input schema, despite
PR #445's from_* naming intervention + system-prompt sister block +
explicit Level 2 memory documenting the anti-pattern.

The fix is a small pure filter that drops every block where
`type == "tool_use" && name.start_with?("from_")` from the API
response's `content` array, applied at the entry point of the main
loop's response handler. The model's tool_use is removed before
`normalize_tool_uses` and `extract_text` run, so:

* a from_*-only response falls through to `response_complete!`
  (clean end of turn — no failed tool_result, no error round-trip,
  no token waste),
* a response with mixed legitimate + from_* tool calls dispatches
  only the legitimate ones,
* a response with text + from_* tool calls keeps the text and ends
  the turn without dispatch.

The filter (`Aoide::PhantomCallFilter.call`) is a pure hash → hash
function with no I/O, AR, or event coupling, so it specs in
isolation against fixture payloads. It runs in
`LLMResponseHandler#emit` immediately *after* `log_raw_response`, so
the dev-only diagnostic log from PR #485 keeps observing the
pre-filter API response — letting us continue to measure
hallucination rate post-fix and decide later whether a feedback loop
(synthesised tool_result, Mneme nudge) is worth adding.

Three integration edge cases covered in the handler spec, plus 13
unit specs against the filter directly (including symbol-keyed
content, missing content keys, malformed blocks).

Closes #480
Self-review surfaced one real subtle bug and one boundary case worth
locking explicitly:

* `lib/aoide/phantom_call_filter.rb`: `response.merge("content" => filtered)`
  on a symbol-keyed input produced a hash containing BOTH `:content`
  (with the phantom block intact) AND `"content"` (filtered). Downstream
  callers were saved by `content_blocks`'s `response["content"] ||
  response[:content]` precedence, but the post-filter hash itself was
  inconsistent — any caller that iterated keys directly would see
  duplicates with conflicting content. Filter now detects the input's
  key style and rewrites at the same key.

* `spec/lib/aoide/phantom_call_filter_spec.rb`: rewrote the symbol-key
  test to assert (a) the result is sanitized at the symbol key and (b)
  no string-key copy was added. Also added an explicit boundary case
  for `name: "from_"` (the bare prefix) — the implementation already
  handled it via `start_with?`, the test now documents the intent.

Test/lint state: 14 filter specs, 16 handler specs (30 total) — 0
failures. standardrb clean. reek clean on the filter.
@hoblin hoblin self-assigned this Apr 30, 2026
@hoblin
hoblin marked this pull request as ready for review April 30, 2026 08:33
@hoblin
hoblin requested a review from bonk-moltbot as a code owner April 30, 2026 08:33

@bonk-moltbot bonk-moltbot left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🐭✅ From spike to fix — measure, then cut.

PhantomCallFilter is exactly the right shape: pure function, hash in → hash out, no AR, no events, no I/O. Immutable — .merge instead of mutating content in-place, original response untouched. 13 unit specs cover every edge: symbol keys, nil content, bare prefix, non-hash blocks, multiple phantoms, mixed real+phantom.

Placement is surgical: after log_raw_response (diagnostic data preserved pre-filter) but before normalize_tool_uses (no from_* enters dispatch). The logging spec updated accordingly — raw blocks still show from_*, but no dispatching tool=from_* lines anymore.

Three edge cases from the ticket explicitly covered in integration specs:

  • from_*-only → clean idle, no failed tool_result, no wasted tokens
  • from_* + legitimate → only legitimate dispatches, one tool_call persisted
  • text + from_* → text kept as agent_message, session idles cleanly

The filter keeps hallucination rate observable (raw blocks in aoide.log) while eliminating the symptom (UnknownToolError, wasted round-trips). Future feedback loop (synthesised tool_result, Mneme nudge) remains a clean extension point.

@bonk-moltbot
bonk-moltbot merged commit 448f994 into main Apr 30, 2026
7 checks passed
@bonk-moltbot
bonk-moltbot deleted the fix/480-strip-phantom-tool-calls branch April 30, 2026 08:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Research spike: source of from_* tool calls in the main loop

2 participants