feat: builder-only seed + mixed lv1/lv2 calibration for outer loop evolution - #1275
Draft
akashgit wants to merge 21 commits into
Draft
feat: builder-only seed + mixed lv1/lv2 calibration for outer loop evolution#1275akashgit wants to merge 21 commits into
akashgit wants to merge 21 commits into
Conversation
…lity, adaptive timeout Merge outer loop code from fix/outer-loop-v1-postmortem branch and implement all Phase 4 code gaps from issue #1274. Part A — Merge outer loop code (15 files, ~4000 lines) Part B — Code fixes: - Document ThreadPoolExecutor as correct for I/O-bound Docker calls - Add optional LLM crossover (crossover_fn parameter + llm_crossover_prompt helper) - Add CLI entry points: factory outer-loop calibrate/evolve Part C — Crash resilience: - CheckpointData Pydantic model with atomic writes (write .tmp then rename) - Checkpoint saved after every generation with full population state - Resume from latest checkpoint via --resume flag Part D — Observability: - ProgressTracker writes append-only JSONL to .factory/outer_loop/progress.jsonl - Events: generation_start/complete, agent_start/complete, eval_start/complete, checkpoint_saved, timeout Part E — Timeout awareness: - Adaptive timeout in DirectFeatureBenchEvaluator: retry once with 2x timeout - Configurable agent timeout via --timeout CLI flag - All timeouts logged with structlog.warning Also adds Workflow.to_dict()/from_dict() for round-trip serialization, registers outer-loop mode in CEO_MODES/RUN_MODES/workflow registry. All 308 outer loop tests pass + 198 smoke tests pass. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sentrux Quality ReportAbsoluteDiff (vs base branch) |
Remove unused imports and variables in tests/test_outer_loop/ (8 ruff F401/F841 errors) and update test_register_all_count from 35 to 36 after outer-loop workflow registration. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Codecov Report❌ Patch coverage is ❌ Your patch check has failed because the patch coverage (78.48%) is below the target coverage (79.00%). You can increase the patch coverage or adjust the target coverage. Additional details and impacted files@@ Coverage Diff @@
## main #1275 +/- ##
==========================================
- Coverage 86.26% 85.63% -0.63%
==========================================
Files 211 229 +18
Lines 23123 25173 +2050
Branches 3656 3980 +324
==========================================
+ Hits 19946 21557 +1611
- Misses 2331 2687 +356
- Partials 846 929 +83 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The featurebench specs are at featurebench/featurebench/<task_id>/ but the evaluator and CLI were looking at featurebench/<task_id>/. Also fix the seed workflow prompts to reference task-instruction.md relative to the project root instead of /tmp/task-instruction.md, since agents run on the host with the testbed as their project directory. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The factory agent CLI doesn't accept --disallowedTools (it's a claude CLI flag), causing all agents to exit with code 2. Also fix patch paths to use .resolve() so git apply can find them when cwd is the testbed. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Orchestrates smoke test → calibration → evolution → report pipeline. Smoke test verified: packaging instance resolved (score=1.0, 280s). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The seed workflow's builder had timeout=7200 (2hrs) which made calibration extremely slow. Reduced to 600s. Also changed the evaluator to use min(node.timeout, agent_timeout) instead of just node.timeout, ensuring the evaluator's agent_timeout acts as a cap. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Standalone scripts for running evolution from calibration data and generating the outer loop v2 report. Calibration running (4/10 done, all passing so far). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Calibration in progress. Instances resolved so far: pydantic (518s), fastapi (242s), pandas (307s), seaborn (319s). Sphinx running. Seed workflow showing strong baseline performance at 100% resolve rate on tested instances. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Calibration running in background (sphinx builder taking >10min due to old 7200s timeout — fixed in code but this run uses pre-fix timeout). All 4 completed instances resolved: pydantic, fastapi, pandas, seaborn. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remaining 4 instances (sympy, mlflow, pytest, packaging) all resolved. Training/holdout split: 7/3. Ready for evolution. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous 4-node seed (researcher→builder→health_checker→gate) scored 100% on all 10 calibration instances, leaving zero variance for evolution. Add create_seed_workflow(minimal=True) that returns a single builder node with no prior codebase study. Expected ~40-60% pass rate, giving the evolutionary search room to discover beneficial mutations (adding researcher nodes, verifiers, etc). Also update scripts/run_evolution.py with CLI args for --minimal, --generations, --population, --budget, --timeout, --parallelism. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lv2 instances have empty setup_patch — agent builds from scratch. Verification uses test.sh flow: pip install agent package, restore original from backup, apply test_patch (import rewriting), run tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ation lv2 agents start from empty testbed (not the base image's solution). Verification now creates backup tar from base image, copies test_patch, and runs test.sh inside container for proper lv2 flow. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- run_calibration_lv2.py: calibrates builder-only seed on 10 lv2 instances - run_evolution_lv2.py: runs SwarmEngine evolution on lv2 with configurable params - generate_report_lv2.py: generates markdown report with per-instance comparison Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Calibrated the builder-only (minimal) seed on the same 10 lv1 instances. Result: 10/10 resolved (100%), same as the 4-node pipeline. Even without a researcher node for prior codebase study, the builder alone solves all lv1 tasks. Pivoting to lv2 instances for meaningful evolutionary signal. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…0.0) All 10 lv2 instances fail with the builder-only seed. Common failure pattern: agent creates code as `agent_code` module but test.sh expects it importable at test collection time with network disabled. Training: 7 instances, Holdout: 3 instances. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lv2 FeatureBench instances (multiple functions per task) are genuinely hard: builder-only seed scored 0% (vs 100% on lv1). This gives evolution a different challenge — can mutations (NODE_INSERT, PROMPT_MUTATE) improve from 0%? Updated generate_report.py to handle lv2 calibration data and compare lv1 vs lv2 results. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lv1=100%, lv2=0% — both have zero variance. Mixed calibration uses 4 lv1 (PASS) + 3 lv2 (FAIL) for training, giving ~57% seed score. Evolution can now discover mutations that help solve lv2 tasks (adding researcher node, improving builder prompt, etc). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…gress Key findings: - lv1: Both 4-node and builder-only seeds score 100% (zero variance) - lv2: Builder-only scores 0% (too hard, all 10 instances FAIL) - Mixed lv1+lv2 evolution running to find optimal workflow Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete calibration results: - lv1: 100% for both 4-node and builder-only (zero variance) - lv2: 0% for builder-only (too hard) - Mixed lv1+lv2: 57% seed score with meaningful variance Key finding: designer's 3-node variant (researcher+builder+health_checker) scores identical to builder-only. The builder prompt is the bottleneck, not the workflow graph structure. Evolution running with mixed calibration, results pending. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… events Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1274
Changes
Builder-only seed workflow: Added
create_seed_workflow(minimal=True)— single-node workflow (just builder, no researcher/health_checker/gate). Gives evolution room to discover mutations.Calibration findings:
Key insight: Designer's 3-node variant (researcher→builder→health_checker) scores identical to builder-only (0.571). Prompt quality > graph structure.
Evolution in progress with mixed calibration (budget=20, 2 generations)
Files
factory/outer_loop/harbor_evaluator.py—create_seed_workflow(minimal=True)+_create_builder_only_seed()scripts/run_evolution.py— CLI args for--minimal,--generations, etcscripts/run_calibration_v2.py— lv1 builder-only calibrationscripts/run_calibration_lv2.py— lv2 builder-only calibrationscripts/run_evolution_mixed.py— mixed lv1+lv2 evolution runnerscripts/generate_report.py— multi-level report generatorresults/outer_loop_v2_report.md— comprehensive report