Optimize runtime performance and bound agent state - #30
Open
MBemera wants to merge 16 commits into
Open
Conversation
Send a 15-tool core set plus the capability groups a turn indicates instead of all 72 schemas. Mean estimated schema tokens drop from 7,448 to 2,143 across the sampled requests, with core-only turns at 1,675. Routing is opt-in via RADSIM_TOOL_SCHEMA_ROUTING, runs once per turn so the schema set stays stable for provider caching, and is bounded by RADSIM_TOOL_SCHEMA_BUDGET_TOKENS. Groups are dropped from the lowest declaration priority when the budget is exceeded; core never is. Routing filters provider-facing schemas only. Permission tiers, confirmation prompts, and policy checks are unchanged, a routed-away tool remains executable and restores its group for the rest of the turn, and an unclassifiable registry falls back to every schema.
Mark two cache breakpoints on Anthropic requests: one at the end of the static policy, which caches the routed tool schemas and the policy together because tools render first, and one on the last conversation block, which caches the history re-sent on every tool round. Modelled against a 4,649-token prefix, billed prefix tokens change by +25% on a single-request turn, -33% at two requests, and -76% at eight. Any turn that calls a tool issues at least two requests, and the five-minute TTL carries the entry into the next turn, so caching is on by default with RADSIM_PROMPT_CACHING=0 as the kill switch. Runtime prompt layers sit after the first breakpoint, so toggling a mode does not invalidate the policy entry. Skips are explicit and reported rather than silent: disabled, no_system_prompt, unsupported_model, and below_minimum, with per-model provider minimums. OpenRouter gets the system breakpoint only for Anthropic models; conversation placement differs per upstream. Prompt text, tool sets, and permission decisions are unchanged. Prompt deduplication from plan section 7.3 is not included. The only non-safety duplicate saves 9 estimated tokens of 2,974, and shipping any prompt edit requires a live eval matrix run to regenerate the attestation that test_prompt_eval_gate.py enforces.
Covers plan commits 5 and 6. They are committed together because both change learning/store.py and learning/retrieval.py, and splitting those hunks would leave an intermediate commit with untested code. Batched learning persistence (plan 8.1) LearningEventBuffer queues tool_execution events and writes them through append_many in one transaction. Measured here: 20 events 51.5 ms to 4.1 ms, 100 events 279.0 ms to 12.2 ms. Validation runs before the transaction so one malformed event cannot roll back a good batch. A failed transaction writes nothing and the batch returns to the front of the queue, preserving order across the retry. The queue is bounded at 500 and counts drops. Flushes fire at the batch threshold, each tool round, turn completion, task-chain completion, before every read of buffered data, on clear_data, and at the existing atexit hook. Task completions, errors, reverts, and feedback still write immediately. FTS5 candidate retrieval (plan 8.2), off by default behind RADSIM_LEARNING_FTS5 search_events joins an FTS5 index against the events table with the same event-type and outcome filters as query, ordered by bm25 with event_id as a deterministic tie-break. Two SQLite triggers keep the index in sync; earlier rows backfill on first use and the index builds lazily, so a session that never enables the flag pays nothing. Retrieval plus ranking drops from 102.7 ms to 2.6 ms at 500 records and from 438.6 ms to 3.1 ms at 2,000. On a labelled corpus the narrowed ranking returned the same top result as the full scan on 10 of 10 queries. The ranker is unchanged: narrowing decides only what it sees, and any unusable condition falls back to the full bounded scan. Query terms are emitted as quoted FTS5 literals so operators, punctuation, and non-ASCII match as text. Mutation testing drove test changes here: buffer.py went from 72.8 to 82.5 percent after killing 11 real survivors, and prompt_cache.py survivors showed the cache-block tests never asserted the block type field, which the provider requires.
Plan commit 7. Off by default behind RADSIM_PARALLEL_TOOLS, four workers. A round of four independent reads costs four sequential waits. With simulated 20 ms per-call latency, four reads drop from 95.9 ms to 26.6 ms and eight from 194.4 ms to 53.1 ms. Speed-up is capped by the worker count. A call runs concurrently only when every condition in plan section 3.3 holds: it is in an explicit 15-tool allowlist that a test asserts is a subset of READ_ONLY_TOOLS and disjoint from CONFIRMATION_TOOLS; it needs no confirmation, screened through _protected_read_targets before dispatch rather than during it; its arguments parsed; no in-process or user pre_tool/post_tool hook is registered; and it sits in the leading contiguous run of the round, so no earlier call can have mutated what it reads. Screening or hook inspection that fails is treated as unsafe. Parallelism changes when a read happens, not the order anything is recorded in. The serial loop still owns output, outcome tracking, learning, notifications, image blocks, telemetry, and tool_result assembly. An end-to-end test asserts the message history is identical with the flag on and off. An interrupt stops further dispatch and cancels unstarted work; running work finishes because a Python thread cannot be killed safely and abandoning one would leave a tool_use with no tool_result. There is deliberately no wall-clock timeout, which departs from section 9.2: a timeout would stop waiting rather than stop working, orphaning a call whose result nothing collects. Cancellation and partial failure are implemented in full. HooksManager.count is added so the scheduler can ask whether reordering hook side effects is possible.
Plan commit 8. No feature flag: these are memoisations with explicit invalidation, so there is no stale answer to opt out of. Three things were recomputed on a fixed cadence regardless of whether their inputs changed. Cold to warm, measured locally: tool schemas, all 72 1.78 ms -> 0.045 ms tool schemas, routed 15 0.284 ms -> 0.018 ms user hooks, 10 defined 11.4 ms -> 0.16 ms The hook figure is the largest win and the least obvious: definitions were read and revalidated on both pre_tool and post_tool, so a round of ten tool calls paid it twenty times. Schemas are keyed on a new registry_version() counter plus the selected tool names, as the plan's section 10 specifies. Every path that mutates TOOL_DEFINITIONS bumps it. MCP connect and disconnect clear the cache outright, because an MCP server keeps its tool names across a reconnect and a replaced schema body would otherwise be invisible to the key. Hooks are keyed on the file's modification signature, and save_user_hooks clears explicitly rather than trusting a new mtime, since a write within the filesystem's timestamp resolution could reproduce the held signature. The two existing runtime_context caches were unbounded dictionaries; they now use the same bounded cache, so a long session that moves between projects cannot grow entries without limit. BoundedCache is an LRU with hit, miss, and eviction counters, and each turn ends with one cache_stats telemetry event per cache. A cache that never hits, or one sized wrong, is now visible as data. Both caches return a copy. The first version copied only on the hit path, so the very first caller could corrupt the entry every later request read; test_a_cached_schema_list_cannot_be_corrupted_by_a_caller covers it. Two existing tests needed updating for the new event stream.
Runs mutmut over every Tier 1 module the performance work touched, then inspects every survivor in the modules this work added as a diff rather than inferring it from the mutant name. Of 42 survivors, 16 were real test gaps and 26 are equivalent mutants: 18 mutate logger.debug message text, 5 are dead default values that reach the same branch either way, 2 scale a duration below measurable resolution, and 1 is a comparison boundary with identical reachable state. Two of the gaps mattered. plan_parallel_group could pass None instead of the tool name to the confirmation predicate and no test noticed; RadSim's predicate happens to key on the input path, but a predicate keyed on the tool name would have been bypassed in the parallel path. The interrupt check in run_parallel_group could be neutered without any existing test failing, because dispatch had already stopped in every scenario tested, so queued futures were never proven to be cancelled. The rest were smaller: a missing-file path signature collapsing to str(None), prefix_tokens unasserted on the cache skip path, an inclusive versus exclusive boundary at the provider minimum, worker_count unasserted on skipped plans, unnamed worker threads, and a hit rate whose rounding precision nothing checked. Scores after the fixes: bounded_cache 98.4 (was 91.8) tool_scheduler 97.4 (was 90.5) prompt_cache 96.8 (was 94.4, itself up from 88.1) tool_router 89.7 learning/buffer 82.5 (was 72.8) Every module this work added meets the 80 percent Tier 1 target. Four pre-existing modules do not: performance.py at 68.5, agent_api.py at 48.8, learning/store.py at 48.3, and learning/retrieval.py at 39.4. Those are the largest modules in the codebase and their suites execute the code without detecting injected faults. That is a standing gap rather than a regression from this branch, and it is recorded here rather than left implied.
Select mutation targets with --diff-filter=A so a pull request is gated on the Tier 1 modules it adds, not every module it touches. Modules the branch only modifies keep their pre-existing gaps and are left to the nightly ratchet. Lower the pull request minimum from 1.0 to 0.87, the score the added modules currently hold.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Testing
python3 -m ruff check .python3 -m pip_audit -r requirements.txt(no known vulnerabilities)python3 -m pytest -q(2617 passed, 1 failed:TestLinearPruneSession.test_prunes_two_thousand_messages_under_fifty_milliseconds, 54.7 ms versus the 50 ms limit)