#480 fix: silently strip from_* tool calls before dispatch#487
Conversation
Issue #480's research spike (PR #485) confirmed the spurious from_* tool calls — `from_shell-runner`, `from_zero-width-sleuth`, etc., that produce `Tools::UnknownToolError` — are LLM hallucinations, not internal leaks. The model emits them as real tool_use blocks in the Anthropic response, complete with a fabricated input schema, despite PR #445's from_* naming intervention + system-prompt sister block + explicit Level 2 memory documenting the anti-pattern. The fix is a small pure filter that drops every block where `type == "tool_use" && name.start_with?("from_")` from the API response's `content` array, applied at the entry point of the main loop's response handler. The model's tool_use is removed before `normalize_tool_uses` and `extract_text` run, so: * a from_*-only response falls through to `response_complete!` (clean end of turn — no failed tool_result, no error round-trip, no token waste), * a response with mixed legitimate + from_* tool calls dispatches only the legitimate ones, * a response with text + from_* tool calls keeps the text and ends the turn without dispatch. The filter (`Aoide::PhantomCallFilter.call`) is a pure hash → hash function with no I/O, AR, or event coupling, so it specs in isolation against fixture payloads. It runs in `LLMResponseHandler#emit` immediately *after* `log_raw_response`, so the dev-only diagnostic log from PR #485 keeps observing the pre-filter API response — letting us continue to measure hallucination rate post-fix and decide later whether a feedback loop (synthesised tool_result, Mneme nudge) is worth adding. Three integration edge cases covered in the handler spec, plus 13 unit specs against the filter directly (including symbol-keyed content, missing content keys, malformed blocks). Closes #480
Self-review surfaced one real subtle bug and one boundary case worth
locking explicitly:
* `lib/aoide/phantom_call_filter.rb`: `response.merge("content" => filtered)`
on a symbol-keyed input produced a hash containing BOTH `:content`
(with the phantom block intact) AND `"content"` (filtered). Downstream
callers were saved by `content_blocks`'s `response["content"] ||
response[:content]` precedence, but the post-filter hash itself was
inconsistent — any caller that iterated keys directly would see
duplicates with conflicting content. Filter now detects the input's
key style and rewrites at the same key.
* `spec/lib/aoide/phantom_call_filter_spec.rb`: rewrote the symbol-key
test to assert (a) the result is sanitized at the symbol key and (b)
no string-key copy was added. Also added an explicit boundary case
for `name: "from_"` (the bare prefix) — the implementation already
handled it via `start_with?`, the test now documents the intent.
Test/lint state: 14 filter specs, 16 handler specs (30 total) — 0
failures. standardrb clean. reek clean on the filter.
bonk-moltbot
left a comment
There was a problem hiding this comment.
🐭✅ From spike to fix — measure, then cut.
PhantomCallFilter is exactly the right shape: pure function, hash in → hash out, no AR, no events, no I/O. Immutable — .merge instead of mutating content in-place, original response untouched. 13 unit specs cover every edge: symbol keys, nil content, bare prefix, non-hash blocks, multiple phantoms, mixed real+phantom.
Placement is surgical: after log_raw_response (diagnostic data preserved pre-filter) but before normalize_tool_uses (no from_* enters dispatch). The logging spec updated accordingly — raw blocks still show from_*, but no dispatching tool=from_* lines anymore.
Three edge cases from the ticket explicitly covered in integration specs:
from_*-only → clean idle, no failed tool_result, no wasted tokensfrom_*+ legitimate → only legitimate dispatches, one tool_call persisted- text +
from_*→ text kept as agent_message, session idles cleanly
The filter keeps hallucination rate observable (raw blocks in aoide.log) while eliminating the symptom (UnknownToolError, wasted round-trips). Future feedback loop (synthesised tool_result, Mneme nudge) remains a clean extension point.
Summary
Implements the fix for #480 based on the research-spike findings recorded in the comment on the issue.
The spike (PR #485) confirmed that spurious
from_*tool calls —from_shell-runner,from_zero-width-sleuth,from_melete_goal, etc. — are LLM hallucinations, not internal leaks. The model emits them as legitimatetool_useblocks in the raw Anthropic response, complete with fabricated input schemas, despite thefrom_*naming convention from PR #445 + system-prompt sister block + a Level 2 memory explicitly documenting the anti-pattern. Three reproductions in ~3 minutes of a single deliberately-constructed session.What this PR does
A small pure filter —
Aoide::PhantomCallFilter.call(hash) → hash— drops every block wheretype == "tool_use" && name.start_with?("from_")fromresponse["content"]before the rest of the main loop sees it. The filter has no AR, no events, no I/O — pure hash transformation, easy to spec in isolation.It runs at the top of
Events::Subscribers::LLMResponseHandler#emit, immediately afterlog_raw_responseso the dev-only diagnostic log from PR #485 keeps observing the pre-filter API response. This matters: we want to keep measuring hallucination rate after the fix lands, and decide later whether a feedback loop (synthesised tool_result, Mneme nudge) is worth adding.After the strip, the existing handler logic carries the rest of the work unchanged:
normalize_tool_usesoperates on the filtered content → nofrom_*enters dispatch.extract_textis unaffected by the strip → text always survives.if tool_uses.any? … elsif session.may_response_complete? …branch handles the empty case naturally →from_*-only responses land at cleanresponse_complete!.Three edge cases the user explicitly asked to cover
from_*tool callstool_result, session →idle(clean end of turn)from_*+ legitimate tool callsexecuting; onetool_callmessage persistedfrom_*tool callsagent_message; no dispatch; session →idleAll three are asserted in
spec/lib/events/subscribers/llm_response_handler_spec.rb. The filter itself has 13 unit specs against fixture payloads (including symbol-keyed content, missing content key, non-array content, malformed blocks).Diagnostic continuity
log/aoide.logkeeps logging the raw API response and rawtool_useblocks pre-filter. Post-fix,dispatching tool=from_*lines should disappear, butfrom_*blocks may still appear inraw tool_use blocks:— that's the signal that the model is still hallucinating. If frequency stays high we'll know without re-instrumenting.Test plan
bundle exec rspec spec/lib/aoide/phantom_call_filter_spec.rb— 13 examples, 0 failuresbundle exec rspec spec/lib/events/subscribers/llm_response_handler_spec.rb— 16 examples (3 new edge cases), 0 failuresbundle exec rspec spec/jobs/drain_job_spec.rb— 11 examples, 0 failures (sanity on upstream emitter)bundle exec standardrb— clean on changed filesbundle exec reek lib/aoide/phantom_call_filter.rb lib/events/subscribers/llm_response_handler.rb— 2 pre-existing FeatureEnvy warnings remain, none introduceddispatching tool=from_*lines no longer appear and noUnknownToolErrorlands in the conversation. (Doing right after this PR opens.)Closes #480