fix: error on ipython calls missing 'code' instead of silent no-op - #129
Open
hallerite wants to merge 1 commit into
Open
fix: error on ipython calls missing 'code' instead of silent no-op#129hallerite wants to merge 1 commit into
hallerite wants to merge 1 commit into
Conversation
…lent no-op
The ipython tool read args.get("code", ""), so a call that put the code under
any other key executed nothing and returned an empty result with no error — the
model then hit NameErrors on state it believed it had created. Laguna-S-2.1
does this routinely, emitting the code under 'cmd'. Return a loud, learnable
error naming the wrong keys instead, so the model (or an RL policy) can correct.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
IpythonTool.executereadsargs.get("code", ""), so a tool call that puts the code under any other key executes nothing, returns an empty result, and raises no error. The model then hitsNameErrors on variables it believes it created, and burns turns rediscovering state.Observed routinely during large-scale eval: a model emits the code under
cmd(a{'cmd': ...}cell), and the silent no-op appeared across many rollouts. Under a turn budget it injects failure that lands stochastically later than the malformed call, so it corrupts credit assignment rather than being locally learnable.Fix
If
codeis absent or empty, return a loud error naming the wrong keys ("got keys: 'cmd'. Pass your Python under the 'code' key.") instead of executing an empty string. Now a learnable signal — the model/policy can correct on the next turn, like it already does for the invalid-JSON-args error path.Tests
test_ipython_missing_code_key_errors_not_noop,test_ipython_empty_code_errors. Full suite 114 passed.See also the depth-limit and skill-preload issues filed separately.
🤖 Generated with Claude Code
Note
Low Risk
Localized validation in the ipython tool with clearer errors; no auth, data, or execution-path changes beyond skipping empty runs.
Overview
Fixes silent IPython no-ops when models pass Python under the wrong argument (e.g. Laguna’s
cmdinstead ofcode). Previouslyexecutetreated a missing or emptycodeas"", ran nothing, and returned an empty success—leading to laterNameErrors and poor RL credit assignment.IpythonTool.executenow fails fast with an explicit error that lists the unexpected keys and tells the caller to usecode. Metrics still record anIpythonExecutedevent with zero input size on this path.Adds
test_ipython_missing_code_key_errors_not_noopandtest_ipython_empty_code_errorsplus a small_ipython_ctx()helper for REPL-free error-path tests.Reviewed by Cursor Bugbot for commit 8f865f8. Bugbot is set up for automated code reviews on this repo. Configure here.