optional asyncio.gather concurrency for the per-task loop (MCPBENCH_CONCURRENCY)#38
Open
dzorlu wants to merge 1 commit into
Open
optional asyncio.gather concurrency for the per-task loop (MCPBENCH_CONCURRENCY)#38dzorlu wants to merge 1 commit into
dzorlu wants to merge 1 commit into
Conversation
Adds an opt-in concurrency knob to _run_single_file_benchmark_core's task loop. Behaviour with MCPBENCH_CONCURRENCY unset or =1 is byte-equivalent to the original sequential loop. With higher values (e.g. 4) the loop launches all tasks under an asyncio.Semaphore and drains them via asyncio.as_completed, so metrics still update incrementally as tasks finish in any order. Motivation: each task is a multi-turn LLM rollout taking ~1-30 min, mostly bottlenecked by per-call LLM latency, not by local CPU or MCP server load. Running them serially leaves the LLM endpoint idle most of the time. On a Fleet eval against a hosted Tinker LoRA, this dropped wall-clock from ~80h to ~5h with concurrency=4 and no observed quality regression (task_completion_score, tool_call_success_rate stayed within noise). Safety: - MCP server discovery happens once in _initialize_benchmark; servers_info is read-only inside the loop and safe to share across coroutines. - LLMProvider is a single AsyncOpenAI instance per model_config, designed for concurrent calls (no client-side mutex). - Per-task exceptions are caught and logged so one bad task doesn't kill the whole gather. Default of 1 keeps existing single-task semantics for users who haven't opted in.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an opt-in
MCPBENCH_CONCURRENCYenv var to_run_single_file_benchmark_core's task loop. Default1keeps the original sequential semantics; with higher values the loop wraps tasks inasyncio.Semaphore(N)and drains viaasyncio.as_completed.Why
Each MCP-Bench task is a multi-turn LLM rollout (~1-30 min, dominated by per-call LLM latency). Serial execution leaves the LLM endpoint idle most of the time. On a Fleet eval against a Tinker LoRA endpoint this dropped wall-clock ~80h → ~5h at
MCPBENCH_CONCURRENCY=4, with no quality regression (task_completion_scoreandtool_call_success_ratewithin noise).Why it's safe
_initialize_benchmark;servers_infois read-only inside the loop.LLMProvider's underlyingAsyncOpenAIclient is concurrency-safe.Test plan
MCPBENCH_CONCURRENCY=4 python run_benchmark.py --models <m> --tasks-file <f>, confirm wall-clock drops linearly and final metrics are within noise of sequential🤖 Generated with Claude Code