feat: terse persona prompt, bash+ipython default tools, actionable compaction handoff - #142
Draft
samsja wants to merge 5 commits into
Draft
feat: terse persona prompt, bash+ipython default tools, actionable compaction handoff#142samsja wants to merge 5 commits into
samsja wants to merge 5 commits into
Conversation
Replace the "general purpose agent / break down into sub-tasks" role framing
and the long IPYTHON_CONTROL_PROMPT ("long-lived notebook ... assign results
to named variables") with a terse coding-agent persona plus one factual
%%bash/project-env line.
Motivation: a 9-model x 8-arm interface/prompt ablation on SWE-rebench (32
tasks/arm, kimi-k3) found the verbose notebook framing - not the ipython
interface, not %%bash syntax - triggers heavy per-turn deliberation in some
models (kimi-k3: 327 reasoning tok/turn and 0.53 solve under the stock prompt
vs 209 tok/turn and 0.73 solve with this persona style, same tools, same
tasks; -20% output tokens). Frontier models were insensitive to the change.
…tion handoff - New native `bash` builtin tool (one command per call, fresh subshell, git guard, output clipping). Default tool set is now bash + ipython (RLM_BUILTIN_TOOLS overrides, e.g. "ipython" for a REPL-only agent). - `edit` builtin skill enabled by default (RLM_SKILLS="" disables). - System prompt names each capability in one line (persona style); the %%bash control line only renders for REPL-only configs, replaced by the project-env rule otherwise. - CHECKPOINT_COMPACTION_PROMPT now demands an actionable handoff: exact repro/test commands and pending edits as fenced code blocks + numbered next steps (validated to make post-compaction resumption concrete: 0 -> ~4 command blocks per handoff, solve-rate neutral-to-positive). Motivation: 32-task interface ablations (kimi-k3) show the bash+ipython+edit-skill config is the best balanced setup (reasoning cost 2nd-lowest of 9 arms tested, full REPL powers retained); tool-availability is otherwise solve-neutral across 9 models. Tests: 112 passed.
samsja
changed the title
- EditTool joins the native tool set; default RLM_BUILTIN_TOOLS = bash, edit, ipython - bash skill added; default RLM_SKILLS = bash, edit — both actions callable from the REPL too (`await bash(...)`, `await edit(...)`) for mixing with Python in one cell or avoiding shell quoting - prompt names each route in one line; edit-skill line reframed as the in-REPL alternative (multiline/quote-heavy payloads) rather than the preferred path Dual-route ablation (32 tasks, kimi-k3): native tools win revealed preference 99:1, solve 0.74 / 199 reasoning tok/turn (best tier); the skill calls that do occur are rational (quoting escape, output separation, py-text-to-shell). Keeping skills registered costs nothing and covers that tail.
"The ipython kernel is an isolated venv without the project's packages — never import project modules there. Everything that executes project code (tests, repros, imports) goes through bash with the project's interpreter." kimi-k3 was importing project modules into the rlm kernel (~25 attempts / 32 rollouts, ~1 in 5 hitting ModuleNotFoundError) despite the old softer wording. Probes (8 tasks x 2 per arm): mechanism-explaining variants drop kernel project-imports 5 -> 0 with zero ImportErrors and no reasoning inflation (157-186 tok/turn vs 182 baseline). This wording is the config-agnostic variant (no skill names or activity examples).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three changes, each from a k3-focused ablation campaign (9 models, 32 tasks/arm, SWE-rebench-V2 easy slice):
bash+ipython,editskill on by default (RLM_BUILTIN_TOOLS/RLM_SKILLSoverride)The starting point: something was wrong
kimi-k3 on stock rlm vs a plain bash+edit agent, same 32 tasks (and SWE-bench Pro, 32 tasks):
Same model, same tasks: stock rlm made k3 reason ~1.5-2x harder per step and solve less. We ablated why.
Interface/prompt ablation (kimi-k3, n=32/arm, same seeded tasks)
!escapesHypotheses tested: ipython-as-sole-interface (falsified - m4/m5a/m6 match bash arms),
%%bashsyntax (falsified - m5a is the best solve arm), the verbose prompt (confirmed - m0 is the only outlier, same tools as m5a/m5b). Non-reasoning output is ~constant (5.1-6.2k) across all arms; the entire cost spread is deliberation. Frontier models (gpt-5.6, sonnet/opus/fable-5, glm, deepseek, luna) are insensitive to all of this - the persona prompt is safe for them and a large win for prompt-sensitive models.Conclusion -> m3 as default: native bash for shell (terse), REPL for logic/skills/state (full powers kept: persistence, skills, subagents), edit as skill. 2nd-cheapest reasoning of all arms; solve differences within persona arms are noise at n=32. m5a (REPL-only) remains a fine config and stays reachable via
RLM_BUILTIN_TOOLS=ipython.Compaction ablation (m3 base, 25k threshold to force compaction, n=32/arm, k3 + luna)
Conclusion -> keep compaction guidance at compaction time only (as this PR does). Advance system-prompt notice is unnecessary: models already stash state naturally and the existing at-compaction REPL note carries variable names into the handoff (k3 reuses pre-compaction variables post-compaction with no advance prompt). The stash line doubles state-keeping but buys no solve. The stock prompt again produced hoarding (29 assigns/rollout) and the worst compacted-solve.
The new
CHECKPOINT_COMPACTION_PROMPTstructure (fenced command blocks + numbered steps) comes from a separate handoff study: it moved k3 handoffs from 0 to ~4 runnable command blocks each, solve neutral-to-positive, making post-compaction resumption concrete instead of prose-only.Tests updated; 112 passed; ruff clean.
Update: bash and edit as both tools AND skills
Both actions are now dual-route: native
bash/edittools for atomic calls, plusawait bash(...)/await edit(...)REPL skills for composing with Python in one cell (output parsing, loops, quote-heavy multiline payloads).Dual-route ablation (m8, 32 tasks, kimi-k3): revealed preference is 99:1 for native tools, solve 0.74 / 199 reasoning tok/turn (best tier of the campaign). The rare skill uses are rational — escaping shell quoting after a failed
python -c, separating outputs of sequential commands, writing a script via Python then executing it. Skills stay registered as a free escape hatch.Update: kernel-isolation line (project imports)
Observed: kimi-k3 imports project modules into the rlm kernel (which has its own venv), hitting
ModuleNotFoundError~1 in 5 attempts (~25 attempts per 32 rollouts) despite a softer "run project code through the project's own environment" line. Recovery was fast (switches to bash, no kernel pip-installs) but wasteful.Fix: the project-env line now states the mechanism and routing explicitly — "the ipython kernel is an isolated venv without the project's packages... everything that executes project code goes through bash with the project's interpreter."
Probe validation (8 tasks x 2 per arm, m8 base): kernel project-imports 5 -> 0, ImportErrors 1 -> 0, reasoning tok/turn 157-186 vs 182 baseline (no inflation; the informational phrasing pattern again). A no-activity-examples variant (this wording) is running for final confirmation; role-list variants performed equally but couple the prompt to specific skills/activities, so the clean variant ships.