diff --git a/docs/en/2.Development Guide/Advanced Usage/Multi-Rollout Parallel Task Execution.md b/docs/en/2.Development Guide/Advanced Usage/Multi-Rollout Parallel Task Execution.md new file mode 100644 index 000000000..3f7c22def --- /dev/null +++ b/docs/en/2.Development Guide/Advanced Usage/Multi-Rollout Parallel Task Execution.md @@ -0,0 +1,113 @@ +# Multi-Rollout Parallel Task Execution + +Spawn N independent agent attempts for the same task, run them in parallel with different strategies, and return the best result. + +## Problem + +A single agent trajectory may get stuck in a local optimum. When faced with a hard bug fix, the first approach the agent tries might not be the best one. Re-trying with a different strategy requires restarting the entire task, which is slow and wasteful. + +## Solution + +**Multi-rollout** generates N isolated execution attempts from the same starting point: + +1. Clone the agent workspace into N isolated sub-workspaces. +2. Inject a different strategy prompt into each attempt. +3. Run all attempts in parallel (each is a full `DeepAgent.invoke()`). +4. Collect the N results. +5. Run a selector to pick the best one. +6. Return the winner. + +## Configuration + +Add `multi_rollout` to your `DeepAgentConfig`: + +```python +from openjiuwen.harness import DeepAgentConfig, MultiRolloutConfig + +config = DeepAgentConfig( + # ... other fields ... + multi_rollout=MultiRolloutConfig( + enabled=True, + n_rollouts=3, + max_parallel=3, + timeout_per_rollout=600.0, + selector_kind="first_successful", + ) +) +``` + +| Field | Default | Description | +|-------|---------|-------------| +| `enabled` | `False` | Turn multi-rollout on/off | +| `n_rollouts` | `3` | Number of parallel attempts | +| `max_parallel` | `0` | Max concurrent rollouts (`0` = unlimited) | +| `timeout_per_rollout` | `600.0` | Timeout per attempt in seconds | +| `selector_kind` | `"first_successful"` | How to pick the winner | + +## Strategy Variants + +Each attempt receives the same task but prefixed with a different strategy instruction. The default three strategies are: + +1. **Correctness-focused** — explore deeply, consider all implications +2. **Minimal-diff** — change as few lines as possible +3. **Edge-case-focused** — consider boundaries, errors, defensive code + +You can replace them: + +```python +config = MultiRolloutConfig( + strategy_variants=[ + "Focus on speed. Get a working fix quickly.", + "Focus on robustness. Handle every edge case.", + "Focus on minimal changes. Preserve existing style.", + ] +) +``` + +## Result Selectors + +| Selector | Behavior | Best for | +|----------|----------|----------| +| `first_successful` | Return the first non-error result | Speed; safest default | +| `longest_output` | Return the successful result with longest output | When completeness matters | +| `shortest_output` | Return the successful result with shortest output | When minimal diffs matter | + +## How It Works + +```python +# When DeepAgent.invoke() is called: +if multi_rollout.enabled and n_rollouts > 1: + for i in range(n_rollouts): + subagent = parent.create_subagent( + "general-purpose", + subsession_id=f"rollout-{i:03d}" + ) + # Each subagent gets its own isolated workspace + + # Run all subagents in parallel via asyncio.gather + # Apply strategy prefix to each attempt's query + # Select best result via configured selector + return winner +else: + return await normal_invoke() +``` + +## Using Without DeepAgentConfig + +You can also use `MultiRolloutExecutor` directly: + +```python +from openjiuwen.harness.multi_rollout import MultiRolloutExecutor, MultiRolloutConfig + +executor = MultiRolloutExecutor(parent_agent, MultiRolloutConfig( + enabled=True, + n_rollouts=3, +)) +result = await executor.invoke({"query": "fix bug"}) +``` + +## Caveats + +- **Streaming**: Multi-rollout only works with `invoke()`, not `stream()`. If you need streaming with rollouts, run the executor first, then stream the winning result separately. +- **Cost**: Each rollout consumes full LLM tokens. 3 rollouts is approximately 3x cost. +- **Workspace state**: The parent workspace is untouched. The winning result is returned as text; copying files back to the parent workspace is the caller's responsibility. diff --git a/docs/en/SUMMARY.md b/docs/en/SUMMARY.md index 4e826018b..e3ee38236 100644 --- a/docs/en/SUMMARY.md +++ b/docs/en/SUMMARY.md @@ -66,6 +66,7 @@ - [Security Guardrail](2.Development%20Guide/Advanced%20Usage/Security%20Guardrail.md) - [MCP Tool](2.Development%20Guide/Advanced%20Usage/MCP%20Tool.md) - [Store Plugin Development](2.Development%20Guide/Advanced%20Usage/Store%20Plugin%20Development.md) + - [Multi-Rollout Parallel Task Execution](2.Development%20Guide/Advanced%20Usage/Multi-Rollout%20Parallel%20Task%20Execution.md) - [IntelliRouter](2.Development%20Guide/Advanced%20Usage/IntelliRouter.md) - [API Docs](2.Development%20Guide/API%20Docs/README.md) - [openjiuwen.harness](2.Development%20Guide/API%20Docs/openjiuwen.harness.README.md) diff --git "a/docs/zh/2.\345\274\200\345\217\221\346\214\207\345\215\227/\351\253\230\351\230\266\347\224\250\346\263\225/Multi-Rollout \345\271\266\350\241\214\344\273\273\345\212\241\346\211\247\350\241\214.md" "b/docs/zh/2.\345\274\200\345\217\221\346\214\207\345\215\227/\351\253\230\351\230\266\347\224\250\346\263\225/Multi-Rollout \345\271\266\350\241\214\344\273\273\345\212\241\346\211\247\350\241\214.md" new file mode 100644 index 000000000..183bccaf7 --- /dev/null +++ "b/docs/zh/2.\345\274\200\345\217\221\346\214\207\345\215\227/\351\253\230\351\230\266\347\224\250\346\263\225/Multi-Rollout \345\271\266\350\241\214\344\273\273\345\212\241\346\211\247\350\241\214.md" @@ -0,0 +1,107 @@ +# Multi-Rollout 并行任务执行 + +为同一任务生成 N 个独立的智能体尝试,以不同策略并行运行,返回最优结果。 + +## 问题 + +单一智能体执行路径可能陷入局部最优。面对复杂 bug 修复时,智能体的第一次尝试往往并非最佳方案。手动重试需要重启整个任务,既慢又浪费。 + +## 解决方案 + +**Multi-Rollout** 从同一出发点生成 N 个独立执行尝试: + +1. 将智能体工作空间克隆为 N 个独立子空间 +2. 为每个尝试注入不同策略提示 +3. 并行运行所有尝试(每个都是完整的 `DeepAgent.invoke()`) +4. 收集 N 个结果 +5. 通过选择器挑选最优 +6. 返回胜者 + +## 配置 + +在 `DeepAgentConfig` 中添加 `multi_rollout`: + +```python +from openjiuwen.harness import DeepAgentConfig, MultiRolloutConfig + +config = DeepAgentConfig( + # ... 其他字段 ... + multi_rollout=MultiRolloutConfig( + enabled=True, + n_rollouts=3, + max_parallel=3, + timeout_per_rollout=600.0, + selector_kind="first_successful", + ) +) +``` + +| 字段 | 默认值 | 说明 | +|------|--------|------| +| `enabled` | `False` | 开启/关闭多轮执行 | +| `n_rollouts` | `3` | 并行尝试次数 | +| `max_parallel` | `0` | 最大并发数(`0` = 不限) | +| `timeout_per_rollout` | `600.0` | 每次尝试超时(秒) | +| `selector_kind` | `"first_successful"` | 选择胜者的方式 | + +## 策略变体 + +每个尝试接收相同任务,但前缀策略指令不同。默认三种策略: + +1. **注重正确性** — 深入探索,考虑所有影响 +2. **最小化改动** — 改动行数越少越好 +3. **注重边界情况** — 考虑边界、错误、防御性代码 + +可自定义: + +```python +config = MultiRolloutConfig( + strategy_variants=[ + "Focus on speed. Get a working fix quickly.", + "Focus on robustness. Handle every edge case.", + "Focus on minimal changes. Preserve existing style.", + ] +) +``` + +## 结果选择器 + +| 选择器 | 行为 | 适用场景 | +|--------|------|----------| +| `first_successful` | 返回第一个无错误结果 | 速度优先;最安全的默认 | +| `longest_output` | 返回输出最长的成功结果 | 完整性优先 | +| `shortest_output` | 返回输出最短的成功结果 | 最小化 diff 优先 | + +## 工作原理 + +``` +用户调用 DeepAgent.invoke() + └─ 如果 multi_rollout.enabled 且 n_rollouts > 1: + 创建 N 个子智能体(各带独立工作空间) + 为每个尝试的 query 添加策略前缀 + 并行运行(asyncio.gather) + 选择最优结果 + 返回胜者 + └─ 否则: + 正常单路径执行 +``` + +## 脱离 DeepAgentConfig 直接使用 + +也可直接使用 `MultiRolloutExecutor`: + +```python +from openjiuwen.harness.multi_rollout import MultiRolloutExecutor, MultiRolloutConfig + +executor = MultiRolloutExecutor(parent_agent, MultiRolloutConfig( + enabled=True, + n_rollouts=3, +)) +result = await executor.invoke({"query": "fix bug"}) +``` + +## 注意事项 + +- **流式输出**:Multi-Rollout 仅支持 `invoke()`,不支持 `stream()`。如需流式,先运行选择器,再对胜果单独流式输出。 +- **成本**:每次尝试消耗完整 LLM token。3 次尝试 ≈ 3 倍成本。 +- **工作空间状态**:父工作空间不受影响。返回结果为文本;如需将文件复制回父空间,由调用方负责。 diff --git a/docs/zh/SUMMARY.md b/docs/zh/SUMMARY.md index 5c3be68bb..78c7c91ea 100644 --- a/docs/zh/SUMMARY.md +++ b/docs/zh/SUMMARY.md @@ -67,6 +67,7 @@ - [安全护栏Guardrail](2.开发指南/高阶用法/安全护栏Guardrail.md) - [MCP工具](2.开发指南/高阶用法/MCP工具.md) - [插件开发-存储后端](2.开发指南/高阶用法/插件开发-存储后端.md) + - [Multi-Rollout 并行任务执行](2.开发指南/高阶用法/Multi-Rollout%20并行任务执行.md) - [IntelliRouter智能路由](2.开发指南/高阶用法/IntelliRouter智能路由.md) - [API文档](2.开发指南/API文档/README.md) - [openjiuwen.harness](2.开发指南/API文档/openjiuwen.harness.README.md) diff --git a/openjiuwen/harness/__init__.py b/openjiuwen/harness/__init__.py index 0a0cd88b3..a0d8ccb09 100644 --- a/openjiuwen/harness/__init__.py +++ b/openjiuwen/harness/__init__.py @@ -20,6 +20,8 @@ VisionModelConfig, ) from openjiuwen.harness.workspace.workspace import Workspace + from openjiuwen.harness.multi_rollout.config import MultiRolloutConfig + from openjiuwen.harness.multi_rollout.executor import MultiRolloutExecutor __all__ = [ "DeepAgent", @@ -28,6 +30,8 @@ "DeepAgentConfig", "AudioModelConfig", "VisionModelConfig", + "MultiRolloutConfig", + "MultiRolloutExecutor", "create_deep_agent", "Workspace", ] @@ -75,6 +79,16 @@ def __getattr__(name: str) -> Any: Workspace, ) return Workspace + if name == "MultiRolloutConfig": + from openjiuwen.harness.multi_rollout.config import ( + MultiRolloutConfig, + ) + return MultiRolloutConfig + if name == "MultiRolloutExecutor": + from openjiuwen.harness.multi_rollout.executor import ( + MultiRolloutExecutor, + ) + return MultiRolloutExecutor raise AttributeError( f"module {__name__!r} has no attribute {name!r}" ) diff --git a/openjiuwen/harness/deep_agent.py b/openjiuwen/harness/deep_agent.py index dc6e2f47c..e6e105c18 100644 --- a/openjiuwen/harness/deep_agent.py +++ b/openjiuwen/harness/deep_agent.py @@ -30,6 +30,9 @@ from openjiuwen.core.common.exception.errors import build_error from openjiuwen.core.common.logging import logger from openjiuwen.core.common.security.user_config import UserConfig +from openjiuwen.harness.multi_rollout.executor import ( + MultiRolloutExecutor, +) from openjiuwen.core.context_engine import ContextEngine from openjiuwen.core.context_engine.context.context_utils import ContextUtils from openjiuwen.core.controller.config import ControllerConfig @@ -2492,6 +2495,7 @@ async def invoke( ) invoke_inputs = self._normalize_inputs(inputs) + ctx = AgentCallbackContext(agent=self, inputs=invoke_inputs, session=session) self._invoke_active = True @@ -2501,7 +2505,12 @@ async def invoke( AgentCallbackEvent.BEFORE_INVOKE, AgentCallbackEvent.AFTER_INVOKE, ): - if ( + if self._use_multi_rollout(): + executor = MultiRolloutExecutor( + self, self._deep_config.multi_rollout + ) + result = await executor.invoke(invoke_inputs, session) + elif ( self._deep_config is not None and self._deep_config.enable_task_loop and not self._is_resume_input(invoke_inputs) @@ -2518,6 +2527,18 @@ async def invoke( finally: self._invoke_active = False + def _use_multi_rollout(self) -> bool: + """Return True when the multi-rollout executor should be used. + + Folds the multi-rollout conditions (config present, multi_rollout + configured, enabled, and more than one rollout requested) into one + boolean so the invoke() branch stays a single expression. + """ + cfg = self._deep_config + if cfg is None or cfg.multi_rollout is None: + return False + return cfg.multi_rollout.enabled and cfg.multi_rollout.n_rollouts > 1 + async def stream( self, inputs: Any, diff --git a/openjiuwen/harness/multi_rollout/__init__.py b/openjiuwen/harness/multi_rollout/__init__.py new file mode 100644 index 000000000..8fc6b7c13 --- /dev/null +++ b/openjiuwen/harness/multi_rollout/__init__.py @@ -0,0 +1,22 @@ +# coding: utf-8 +"""Multi-rollout package for parallel task execution.""" + +from openjiuwen.harness.multi_rollout.config import MultiRolloutConfig +from openjiuwen.harness.multi_rollout.executor import MultiRolloutExecutor +from openjiuwen.harness.multi_rollout.selector import ( + FirstSuccessfulSelector, + LongestOutputSelector, + RolloutResult, + ShortestOutputSelector, + get_selector, +) + +__all__ = [ + "FirstSuccessfulSelector", + "LongestOutputSelector", + "MultiRolloutConfig", + "MultiRolloutExecutor", + "RolloutResult", + "ShortestOutputSelector", + "get_selector", +] diff --git a/openjiuwen/harness/multi_rollout/config.py b/openjiuwen/harness/multi_rollout/config.py new file mode 100644 index 000000000..7993c5472 --- /dev/null +++ b/openjiuwen/harness/multi_rollout/config.py @@ -0,0 +1,45 @@ +# coding: utf-8 +# Copyright (c) Huawei Technologies Co., Ltd. 2026. All rights reserved. +"""Multi-rollout configuration.""" + +from __future__ import annotations + +from dataclasses import dataclass, field +from typing import Any + + +@dataclass +class MultiRolloutConfig: + """Configuration for parallel multi-rollout task execution. + + When enabled, the agent spawns *n_rollouts* isolated subagents, + each with a different strategy, runs them in parallel, and returns + the best result. + """ + + enabled: bool = False + n_rollouts: int = 3 + max_parallel: int = 0 # 0 means unlimited (bounded by asyncio) + timeout_per_rollout: float = 600.0 + strategy_variants: list[str] = field( + default_factory=lambda: [ + ( + "Approach: focus on correctness and thoroughness. " + "Explore deeply, consider all implications, and produce " + "a robust solution." + ), + ( + "Approach: focus on minimal changes. Change as few lines " + "as possible while still fixing the issue. Preserve existing " + "structure and style." + ), + ( + "Approach: focus on edge cases and defensive programming. " + "Consider boundary conditions, error handling, and " + "unexpected inputs." + ), + ] + ) + selector_kind: str = "first_successful" + # Optional: extra kwargs passed to the selector + selector_kwargs: dict[str, Any] = field(default_factory=dict) diff --git a/openjiuwen/harness/multi_rollout/executor.py b/openjiuwen/harness/multi_rollout/executor.py new file mode 100644 index 000000000..7c033463c --- /dev/null +++ b/openjiuwen/harness/multi_rollout/executor.py @@ -0,0 +1,250 @@ +# coding: utf-8 +# Copyright (c) Huawei Technologies Co., Ltd. 2026. All rights reserved. +"""Multi-rollout executor — spawn N isolated attempts, run in parallel, +select best result. + +This is the core engine for the task-layer multi-rollout feature. +It wraps a parent DeepAgent, creates N subagents with isolated workspaces, +applies strategy variants, runs them concurrently, and returns the best +result via a pluggable selector. +""" + +from __future__ import annotations + +import asyncio +import logging +import time +from typing import TYPE_CHECKING, Any + +from openjiuwen.harness.multi_rollout.config import ( + MultiRolloutConfig, +) +from openjiuwen.harness.multi_rollout.selector import ( + RolloutResult, + get_selector, +) + +if TYPE_CHECKING: + from openjiuwen.harness.deep_agent import DeepAgent + +logger = logging.getLogger(__name__) + + +class MultiRolloutExecutor: + """Execute a task via N parallel rollouts and return the best result. + + Usage:: + + executor = MultiRolloutExecutor(parent_agent, config) + result = await executor.invoke({"query": "fix bug #123"}) + + The executor is transparent: when multi-rollout is disabled it + delegates directly to the parent agent. + """ + + def __init__( + self, + parent_agent: "DeepAgent", + config: MultiRolloutConfig, + ) -> None: + self._parent = parent_agent + self._config = config + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + def is_enabled(self) -> bool: + return self._config.enabled and self._config.n_rollouts > 1 + + async def invoke( + self, + inputs: Any, + session: Any | None = None, + ) -> dict[str, Any]: + """Run the task with multi-rollout and return the best result. + + Args: + inputs: Standard agent inputs (dict with "query", etc.). + session: Optional parent session. + + Returns: + The selected best result dict (same shape as parent agent output). + """ + if not self.is_enabled(): + return await self._parent.invoke(inputs, session) + + start = time.monotonic() + n = self._config.n_rollouts + logger.info( + "[MultiRollout] Starting %d parallel rollouts", + n, + ) + + # 1. Create isolated subagents + subagents = self._create_subagents(n) + + # 2. Build per-attempt inputs with strategy prefix + attempt_inputs = self._build_attempt_inputs(inputs, n) + + # 3. Execute in parallel with timeout + results = await self._execute_parallel(subagents, attempt_inputs) + + # 4. Select best + selector = get_selector(self._config.selector_kind) + best = selector.select(results) + + elapsed = time.monotonic() - start + if best.is_success: + logger.info( + "[MultiRollout] Selected attempt %d / %d in %.2fs", + best.attempt_index, + n, + elapsed, + ) + else: + logger.warning( + "[MultiRollout] All %d attempts failed in %.2fs", + n, + elapsed, + ) + + # 5. Return best result (unwrap from RolloutResult) + if best.exception is not None: + raise best.exception + return best.result + + # ------------------------------------------------------------------ + # Internal helpers + # ------------------------------------------------------------------ + + def _create_subagents(self, n: int) -> list["DeepAgent"]: + """Create N isolated subagents via the parent's factory.""" + agents: list[DeepAgent] = [] + for i in range(n): + sub_id = f"rollout-{i:03d}" + try: + sub = self._parent.create_subagent( + "general-purpose", + subsession_id=sub_id, + ) + agents.append(sub) + logger.debug( + "[MultiRollout] Created subagent %s", sub_id + ) + except Exception as exc: + logger.warning( + "[MultiRollout] Failed to create subagent %s: %s", + sub_id, + exc, + ) + raise + return agents + + def _build_attempt_inputs( + self, + base_inputs: Any, + n: int, + ) -> list[Any]: + """Inject strategy variant into each attempt's query.""" + variants = self._config.strategy_variants + results: list[Any] = [] + for i in range(n): + inp = self._copy_inputs(base_inputs) + strategy = variants[i % len(variants)] + query = self._extract_query(inp) + if query: + new_query = f"{strategy}\n\nTask:\n{query}" + self._set_query(inp, new_query) + results.append(inp) + return results + + @staticmethod + def _copy_inputs(inputs: Any) -> Any: + """Shallow-copy inputs dict so each attempt is independent.""" + if isinstance(inputs, dict): + return dict(inputs) + return inputs + + @staticmethod + def _extract_query(inputs: Any) -> str: + """Best-effort extraction of the user query from inputs.""" + if isinstance(inputs, dict): + return str(inputs.get("query", inputs.get("content", ""))) + return str(inputs) if inputs is not None else "" + + @staticmethod + def _set_query(inputs: Any, query: str) -> None: + """Set the query field back into inputs.""" + if isinstance(inputs, dict): + if "query" in inputs: + inputs["query"] = query + elif "content" in inputs: + inputs["content"] = query + + async def _execute_parallel( + self, + subagents: list["DeepAgent"], + attempt_inputs: list[Any], + ) -> list[RolloutResult]: + """Run all subagents in parallel with individual timeouts.""" + n = len(subagents) + timeout = self._config.timeout_per_rollout + max_par = self._config.max_parallel or n + + semaphore = asyncio.Semaphore(max_par) + + async def _run_one( + idx: int, + agent: "DeepAgent", + inp: Any, + ) -> RolloutResult: + async with semaphore: + logger.debug( + "[MultiRollout] Attempt %d starting", idx + ) + try: + result = await asyncio.wait_for( + agent.invoke(inp), + timeout=timeout, + ) + logger.debug( + "[MultiRollout] Attempt %d completed", + idx, + ) + return RolloutResult( + result=result, + attempt_index=idx, + ) + except asyncio.TimeoutError: + logger.warning( + "[MultiRollout] Attempt %d timed out " + "after %.1fs", + idx, + timeout, + ) + return RolloutResult( + result=None, + attempt_index=idx, + exception=TimeoutError( + f"Rollout {idx} exceeded {timeout}s" + ), + ) + except Exception as exc: + logger.warning( + "[MultiRollout] Attempt %d failed: %s", + idx, + exc, + exc_info=True, + ) + return RolloutResult( + result=None, + attempt_index=idx, + exception=exc, + ) + + tasks = [ + asyncio.create_task(_run_one(i, subagents[i], attempt_inputs[i])) + for i in range(n) + ] + return await asyncio.gather(*tasks) diff --git a/openjiuwen/harness/multi_rollout/selector.py b/openjiuwen/harness/multi_rollout/selector.py new file mode 100644 index 000000000..07db2e894 --- /dev/null +++ b/openjiuwen/harness/multi_rollout/selector.py @@ -0,0 +1,145 @@ +# coding: utf-8 +# Copyright (c) Huawei Technologies Co., Ltd. 2026. All rights reserved. +"""Result selectors — pick the best result from N rollouts.""" + +from __future__ import annotations + +import logging +from typing import Any, Protocol + +logger = logging.getLogger(__name__) + + +class RolloutResult: + """Wrapper for a single rollout result with metadata.""" + + def __init__( + self, + result: Any, + attempt_index: int, + exception: Exception | None = None, + ): + self.result = result + self.attempt_index = attempt_index + self.exception = exception + + @property + def is_success(self) -> bool: + return self.exception is None and self.result is not None + + @property + def output_text(self) -> str: + """Extract human-readable output for comparison.""" + if self.exception is not None: + return "" + if isinstance(self.result, dict): + # Common result shapes + for key in ("output", "content", "text", "answer"): + if key in self.result: + val = self.result[key] + return str(val) if val is not None else "" + return str(self.result) if self.result is not None else "" + + +class ResultSelector(Protocol): + """Protocol for selecting the best rollout result.""" + + def select(self, results: list[RolloutResult]) -> RolloutResult: + """Return the best result. Must not mutate *results*.""" + ... + + +class FirstSuccessfulSelector: + """Return the first successful (non-exception) result. + + This is the fastest selector and a safe default. + """ + + @staticmethod + def select(results: list[RolloutResult]) -> RolloutResult: + if not results: + raise ValueError("No rollout results to select from") + + for r in results: + if r.is_success: + logger.info( + "[FirstSuccessfulSelector] Selected attempt %d", + r.attempt_index, + ) + return r + + # All failed — return first so caller sees the exception + logger.warning( + "[FirstSuccessfulSelector] All %d attempts failed; " + "returning first for diagnostics", + len(results), + ) + return results[0] + + +class LongestOutputSelector: + """Return the successful result with the longest output text. + + Useful when longer outputs indicate more complete solutions. + """ + + @staticmethod + def select(results: list[RolloutResult]) -> RolloutResult: + if not results: + raise ValueError("No rollout results to select from") + + successes = [r for r in results if r.is_success] + if not successes: + return results[0] + + best = max(successes, key=lambda r: len(r.output_text)) + logger.info( + "[LongestOutputSelector] Selected attempt %d " + "(output_len=%d)", + best.attempt_index, + len(best.output_text), + ) + return best + + +class ShortestOutputSelector: + """Return the successful result with the shortest output text. + + Useful when brevity indicates precision (e.g., minimal diffs). + """ + + @staticmethod + def select(results: list[RolloutResult]) -> RolloutResult: + if not results: + raise ValueError("No rollout results to select from") + + successes = [r for r in results if r.is_success] + if not successes: + return results[0] + + best = min(successes, key=lambda r: len(r.output_text)) + logger.info( + "[ShortestOutputSelector] Selected attempt %d " + "(output_len=%d)", + best.attempt_index, + len(best.output_text), + ) + return best + + +_SELECTORS: dict[str, type[ResultSelector]] = { + "first_successful": FirstSuccessfulSelector, + "longest_output": LongestOutputSelector, + "shortest_output": ShortestOutputSelector, +} + + +def get_selector(kind: str) -> ResultSelector: + """Factory: instantiate a selector by kind name.""" + cls = _SELECTORS.get(kind) + if cls is None: + raise ValueError( + f"Unknown selector kind: {kind!r}. " + f"Available: {list(_SELECTORS.keys())}" + ) + return cls() diff --git a/openjiuwen/harness/schema/config.py b/openjiuwen/harness/schema/config.py index 93c9e49ff..ef18643a7 100644 --- a/openjiuwen/harness/schema/config.py +++ b/openjiuwen/harness/schema/config.py @@ -24,6 +24,9 @@ from openjiuwen.harness.workspace.workspace import ( Workspace, ) +from openjiuwen.harness.multi_rollout.config import ( + MultiRolloutConfig, +) if TYPE_CHECKING: from openjiuwen.harness.deep_agent import DeepAgent @@ -285,6 +288,11 @@ class DeepAgentConfig: # Subagents inherit the stricter of their own spec and this value. restrict_to_work_dir: bool = True + # Multi-rollout: spawn N parallel attempts for the same task. + multi_rollout: "MultiRolloutConfig" = field( + default_factory=lambda: MultiRolloutConfig() + ) + @dataclass class SubAgentConfig: diff --git a/tests/unit_tests/harness/multi_rollout/test_executor.py b/tests/unit_tests/harness/multi_rollout/test_executor.py new file mode 100644 index 000000000..c94a272ac --- /dev/null +++ b/tests/unit_tests/harness/multi_rollout/test_executor.py @@ -0,0 +1,233 @@ +# coding: utf-8 +# Copyright (c) Huawei Technologies Co., Ltd. 2026. All rights reserved. +"""test_multi_rollout — MultiRolloutExecutor 单元测试。""" + +from __future__ import annotations + +from unittest import IsolatedAsyncioTestCase +from unittest.mock import AsyncMock, MagicMock + +from openjiuwen.harness.multi_rollout.config import MultiRolloutConfig +from openjiuwen.harness.multi_rollout.executor import MultiRolloutExecutor +from openjiuwen.harness.multi_rollout.selector import ( + FirstSuccessfulSelector, + LongestOutputSelector, + RolloutResult, + ShortestOutputSelector, + get_selector, +) + + +class TestMultiRolloutConfig(IsolatedAsyncioTestCase): + def test_defaults(self): + cfg = MultiRolloutConfig() + assert cfg.enabled is False + assert cfg.n_rollouts == 3 + assert cfg.max_parallel == 0 + assert cfg.timeout_per_rollout == 600.0 + assert len(cfg.strategy_variants) == 3 + assert cfg.selector_kind == "first_successful" + + def test_enabled_requires_n_rollouts(self): + cfg = MultiRolloutConfig(enabled=True, n_rollouts=1) + executor = MultiRolloutExecutor(MagicMock(), cfg) + assert executor.is_enabled() is False # n_rollouts must be > 1 + + +class TestRolloutResult(IsolatedAsyncioTestCase): + def test_success(self): + r = RolloutResult(result={"output": "hello"}, attempt_index=0) + assert r.is_success is True + assert r.output_text == "hello" + + def test_failure(self): + r = RolloutResult( + result=None, attempt_index=0, exception=RuntimeError("boom") + ) + assert r.is_success is False + assert r.output_text == "" + + def test_extract_keys(self): + r = RolloutResult(result={"content": "c", "query": "q"}, attempt_index=0) + assert r.output_text == "c" # content preferred over query + + +class TestFirstSuccessfulSelector(IsolatedAsyncioTestCase): + def test_selects_first_success(self): + candidates = [ + RolloutResult(result=None, attempt_index=0, exception=RuntimeError("fail")), + RolloutResult(result={"output": "ok"}, attempt_index=1), + RolloutResult(result={"output": "better"}, attempt_index=2), + ] + sel = FirstSuccessfulSelector() + best = sel.select(candidates) + assert best.attempt_index == 1 + + def test_all_failed_returns_first(self): + candidates = [ + RolloutResult(result=None, attempt_index=0, exception=RuntimeError("a")), + RolloutResult(result=None, attempt_index=1, exception=RuntimeError("b")), + ] + sel = FirstSuccessfulSelector() + best = sel.select(candidates) + assert best.attempt_index == 0 + + def test_empty_raises(self): + sel = FirstSuccessfulSelector() + with self.assertRaises(ValueError): + sel.select([]) + + +class TestLongestOutputSelector(IsolatedAsyncioTestCase): + def test_selects_longest(self): + candidates = [ + RolloutResult(result={"output": "short"}, attempt_index=0), + RolloutResult(result={"output": "this is much longer text"}, attempt_index=1), + RolloutResult(result={"output": "mid"}, attempt_index=2), + ] + sel = LongestOutputSelector() + best = sel.select(candidates) + assert best.attempt_index == 1 + + def test_skips_failed(self): + candidates = [ + RolloutResult(result=None, attempt_index=0, exception=RuntimeError("fail")), + RolloutResult(result={"output": "x"}, attempt_index=1), + ] + sel = LongestOutputSelector() + best = sel.select(candidates) + assert best.attempt_index == 1 + + def test_all_failed_returns_first(self): + candidates = [ + RolloutResult(result=None, attempt_index=0, exception=RuntimeError("a")), + ] + sel = LongestOutputSelector() + best = sel.select(candidates) + assert best.attempt_index == 0 + + +class TestShortestOutputSelector(IsolatedAsyncioTestCase): + def test_selects_shortest(self): + candidates = [ + RolloutResult(result={"output": "long text here"}, attempt_index=0), + RolloutResult(result={"output": "s"}, attempt_index=1), + RolloutResult(result={"output": "medium"}, attempt_index=2), + ] + sel = ShortestOutputSelector() + best = sel.select(candidates) + assert best.attempt_index == 1 + + +class TestGetSelector(IsolatedAsyncioTestCase): + def test_known_selectors(self): + assert isinstance(get_selector("first_successful"), FirstSuccessfulSelector) + assert isinstance(get_selector("longest_output"), LongestOutputSelector) + assert isinstance(get_selector("shortest_output"), ShortestOutputSelector) + + def test_unknown_raises(self): + with self.assertRaises(ValueError): + get_selector("nonexistent") + + +class TestMultiRolloutExecutor(IsolatedAsyncioTestCase): + async def test_disabled_delegates_to_parent(self): + parent = MagicMock() + parent.invoke = AsyncMock(return_value={"output": "parent"}) + cfg = MultiRolloutConfig(enabled=False) + executor = MultiRolloutExecutor(parent, cfg) + + result = await executor.invoke({"query": "test"}) + + assert result == {"output": "parent"} + parent.invoke.assert_awaited_once_with({"query": "test"}, None) + parent.create_subagent.assert_not_called() + + async def test_enabled_spawns_and_selects(self): + parent = MagicMock() + parent.invoke = AsyncMock(return_value={"output": "parent"}) + + # Create 3 mock subagents that return different results + subagents = [] + for i in range(3): + sub = MagicMock() + sub.invoke = AsyncMock(return_value={"output": f"sub-{i}"}) + subagents.append(sub) + + parent.create_subagent = MagicMock(side_effect=subagents) + + cfg = MultiRolloutConfig(enabled=True, n_rollouts=3, timeout_per_rollout=5.0) + executor = MultiRolloutExecutor(parent, cfg) + + result = await executor.invoke({"query": "fix bug"}) + + # Should create 3 subagents + assert parent.create_subagent.call_count == 3 + + # Should select first successful (default selector) + assert result == {"output": "sub-0"} + + async def test_picks_first_success_when_some_fail(self): + parent = MagicMock() + + subagents = [] + for i in range(3): + sub = MagicMock() + if i == 0: + sub.invoke = AsyncMock(side_effect=RuntimeError("fail")) + else: + sub.invoke = AsyncMock(return_value={"output": f"sub-{i}"}) + subagents.append(sub) + + parent.create_subagent = MagicMock(side_effect=subagents) + + cfg = MultiRolloutConfig(enabled=True, n_rollouts=3) + executor = MultiRolloutExecutor(parent, cfg) + + result = await executor.invoke({"query": "fix bug"}) + + assert result == {"output": "sub-1"} + + async def test_all_fail_raises(self): + parent = MagicMock() + + subagents = [] + for i in range(2): + sub = MagicMock() + sub.invoke = AsyncMock(side_effect=RuntimeError(f"fail-{i}")) + subagents.append(sub) + + parent.create_subagent = MagicMock(side_effect=subagents) + + cfg = MultiRolloutConfig(enabled=True, n_rollouts=2) + executor = MultiRolloutExecutor(parent, cfg) + + with self.assertRaises(RuntimeError) as ctx: + await executor.invoke({"query": "fix bug"}) + assert "fail-0" in str(ctx.exception) + + async def test_strategy_prefix_injected(self): + parent = MagicMock() + + captured_inputs = [] + + async def mock_invoke(inputs): + captured_inputs.append(inputs) + return {"output": "ok"} + + sub = MagicMock() + sub.invoke = AsyncMock(side_effect=mock_invoke) + + parent.create_subagent = MagicMock(return_value=sub) + + # n_rollouts must be > 1 for multi-rollout to actually run + cfg = MultiRolloutConfig(enabled=True, n_rollouts=2) + executor = MultiRolloutExecutor(parent, cfg) + + await executor.invoke({"query": "fix bug"}) + + assert len(captured_inputs) == 2 + for inp in captured_inputs: + query = inp["query"] + assert "Approach:" in query + assert "fix bug" in query