Skip to content

feat: builder-only seed + mixed lv1/lv2 calibration for outer loop evolution - #1275

Draft
akashgit wants to merge 21 commits into
mainfrom
factory/run-17122260
Draft

feat: builder-only seed + mixed lv1/lv2 calibration for outer loop evolution#1275
akashgit wants to merge 21 commits into
mainfrom
factory/run-17122260

Conversation

@akashgit

@akashgit akashgit commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Closes #1274

Changes

  • Builder-only seed workflow: Added create_seed_workflow(minimal=True) — single-node workflow (just builder, no researcher/health_checker/gate). Gives evolution room to discover mutations.

  • Calibration findings:

    • lv1: Both 4-node and builder-only seeds score 100% — zero variance
    • lv2: Builder-only scores 0% — too hard for any workflow variant
    • Mixed lv1+lv2: 57% seed score with meaningful variance
  • Key insight: Designer's 3-node variant (researcher→builder→health_checker) scores identical to builder-only (0.571). Prompt quality > graph structure.

  • Evolution in progress with mixed calibration (budget=20, 2 generations)

Files

  • factory/outer_loop/harbor_evaluator.pycreate_seed_workflow(minimal=True) + _create_builder_only_seed()
  • scripts/run_evolution.py — CLI args for --minimal, --generations, etc
  • scripts/run_calibration_v2.py — lv1 builder-only calibration
  • scripts/run_calibration_lv2.py — lv2 builder-only calibration
  • scripts/run_evolution_mixed.py — mixed lv1+lv2 evolution runner
  • scripts/generate_report.py — multi-level report generator
  • results/outer_loop_v2_report.md — comprehensive report

…lity, adaptive timeout

Merge outer loop code from fix/outer-loop-v1-postmortem branch and implement
all Phase 4 code gaps from issue #1274.

Part A — Merge outer loop code (15 files, ~4000 lines)
Part B — Code fixes:
  - Document ThreadPoolExecutor as correct for I/O-bound Docker calls
  - Add optional LLM crossover (crossover_fn parameter + llm_crossover_prompt helper)
  - Add CLI entry points: factory outer-loop calibrate/evolve
Part C — Crash resilience:
  - CheckpointData Pydantic model with atomic writes (write .tmp then rename)
  - Checkpoint saved after every generation with full population state
  - Resume from latest checkpoint via --resume flag
Part D — Observability:
  - ProgressTracker writes append-only JSONL to .factory/outer_loop/progress.jsonl
  - Events: generation_start/complete, agent_start/complete, eval_start/complete,
    checkpoint_saved, timeout
Part E — Timeout awareness:
  - Adaptive timeout in DirectFeatureBenchEvaluator: retry once with 2x timeout
  - Configurable agent timeout via --timeout CLI flag
  - All timeouts logged with structlog.warning

Also adds Workflow.to_dict()/from_dict() for round-trip serialization,
registers outer-loop mode in CEO_MODES/RUN_MODES/workflow registry.

All 308 outer loop tests pass + 198 smoke tests pass.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Aug 15, 2026

Copy link
Copy Markdown

Sentrux Quality Report

Absolute

Scanning ....
[scan] git ls-files: 661 total, 649 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 649 files, 107 unique dirs, 98 cache misses, 5.3ms
[resolve] 1252 resolved, 1671 unresolved (of 2923 total specs)
[resolve_imports] project_map 5.4ms, suffix_idx 1.2ms, suffix_resolve 19.8ms, total 26.4ms
[build_graphs] 649 files | maps 2.5ms, imports 26.7ms, calls+inherit 7.2ms, total 36.4ms | 1251 import, 9797 call, 11 inherit edges
sentrux check — 3 rules checked

Quality: 4700

✗ [Error] max_cc: 3 function(s) exceed max cyclomatic complexity of 30
    factory/cli/_ceo_helpers.py:_validate_ceo_flags (cc=43)
    factory/cli/_ceo_helpers.py:_execute_ceo (cc=43)
    factory/cli/run.py:cmd_run (cc=32)

✗ 1 violation(s) found

Diff (vs base branch)

Scanning ....
[scan] git ls-files: 661 total, 649 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 649 files, 107 unique dirs, 98 cache misses, 5.6ms
[resolve] 1252 resolved, 1671 unresolved (of 2923 total specs)
[resolve_imports] project_map 5.8ms, suffix_idx 1.2ms, suffix_resolve 19.8ms, total 26.8ms
[build_graphs] 649 files | maps 2.4ms, imports 27.0ms, calls+inherit 7.5ms, total 36.9ms | 1251 import, 9797 call, 11 inherit edges
sentrux gate — structural regression check

Quality:      4693 -> 4700
Coupling:     0.79 → 0.79
Cycles:       3 → 3
God files:    3 → 3

Distance from Main Sequence: 0.33

✗ DEGRADED
  ✗ Complex functions increased: 56 → 65

Remove unused imports and variables in tests/test_outer_loop/ (8 ruff
F401/F841 errors) and update test_register_all_count from 35 to 36
after outer-loop workflow registration.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@codecov

codecov Bot commented Aug 15, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.48780% with 441 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.63%. Comparing base (8d94e67) to head (86b5a17).

Files with missing lines Patch % Lines
factory/outer_loop/direct_evaluator.py 21.36% 167 Missing and 6 partials ⚠️
factory/cli/outer_loop.py 11.30% 102 Missing ⚠️
factory/outer_loop/engine.py 84.89% 23 Missing and 22 partials ⚠️
factory/outer_loop/mutations.py 88.63% 21 Missing and 24 partials ⚠️
factory/outer_loop/filesystem.py 79.79% 14 Missing and 6 partials ⚠️
factory/outer_loop/designer.py 85.85% 4 Missing and 10 partials ⚠️
factory/outer_loop/subset.py 86.90% 8 Missing and 3 partials ⚠️
factory/workflow/primitives.py 73.52% 4 Missing and 5 partials ⚠️
factory/outer_loop/run_evolution.py 88.13% 5 Missing and 2 partials ⚠️
factory/outer_loop/harbor_evaluator.py 94.28% 3 Missing and 1 partial ⚠️
... and 5 more

❌ Your patch check has failed because the patch coverage (78.48%) is below the target coverage (79.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1275      +/-   ##
==========================================
- Coverage   86.26%   85.63%   -0.63%     
==========================================
  Files         211      229      +18     
  Lines       23123    25173    +2050     
  Branches     3656     3980     +324     
==========================================
+ Hits        19946    21557    +1611     
- Misses       2331     2687     +356     
- Partials      846      929      +83     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

akashgit and others added 18 commits August 15, 2026 18:07
The featurebench specs are at featurebench/featurebench/<task_id>/ but
the evaluator and CLI were looking at featurebench/<task_id>/. Also fix
the seed workflow prompts to reference task-instruction.md relative to
the project root instead of /tmp/task-instruction.md, since agents run
on the host with the testbed as their project directory.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The factory agent CLI doesn't accept --disallowedTools (it's a claude
CLI flag), causing all agents to exit with code 2. Also fix patch paths
to use .resolve() so git apply can find them when cwd is the testbed.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Orchestrates smoke test → calibration → evolution → report pipeline.
Smoke test verified: packaging instance resolved (score=1.0, 280s).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The seed workflow's builder had timeout=7200 (2hrs) which made
calibration extremely slow. Reduced to 600s. Also changed the
evaluator to use min(node.timeout, agent_timeout) instead of just
node.timeout, ensuring the evaluator's agent_timeout acts as a cap.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Standalone scripts for running evolution from calibration data and
generating the outer loop v2 report. Calibration running (4/10 done,
all passing so far).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Calibration in progress. Instances resolved so far: pydantic (518s),
fastapi (242s), pandas (307s), seaborn (319s). Sphinx running. Seed
workflow showing strong baseline performance at 100% resolve rate
on tested instances.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Calibration running in background (sphinx builder taking >10min due to
old 7200s timeout — fixed in code but this run uses pre-fix timeout).
All 4 completed instances resolved: pydantic, fastapi, pandas, seaborn.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remaining 4 instances (sympy, mlflow, pytest, packaging) all resolved.
Training/holdout split: 7/3. Ready for evolution.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The previous 4-node seed (researcher→builder→health_checker→gate) scored
100% on all 10 calibration instances, leaving zero variance for evolution.

Add create_seed_workflow(minimal=True) that returns a single builder node
with no prior codebase study. Expected ~40-60% pass rate, giving the
evolutionary search room to discover beneficial mutations (adding
researcher nodes, verifiers, etc).

Also update scripts/run_evolution.py with CLI args for --minimal,
--generations, --population, --budget, --timeout, --parallelism.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lv2 instances have empty setup_patch — agent builds from scratch.
Verification uses test.sh flow: pip install agent package, restore
original from backup, apply test_patch (import rewriting), run tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ation

lv2 agents start from empty testbed (not the base image's solution).
Verification now creates backup tar from base image, copies test_patch,
and runs test.sh inside container for proper lv2 flow.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- run_calibration_lv2.py: calibrates builder-only seed on 10 lv2 instances
- run_evolution_lv2.py: runs SwarmEngine evolution on lv2 with configurable params
- generate_report_lv2.py: generates markdown report with per-instance comparison

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Calibrated the builder-only (minimal) seed on the same 10 lv1 instances.
Result: 10/10 resolved (100%), same as the 4-node pipeline. Even without
a researcher node for prior codebase study, the builder alone solves all
lv1 tasks. Pivoting to lv2 instances for meaningful evolutionary signal.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…0.0)

All 10 lv2 instances fail with the builder-only seed. Common failure
pattern: agent creates code as `agent_code` module but test.sh expects
it importable at test collection time with network disabled.

Training: 7 instances, Holdout: 3 instances.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lv2 FeatureBench instances (multiple functions per task) are genuinely
hard: builder-only seed scored 0% (vs 100% on lv1). This gives evolution
a different challenge — can mutations (NODE_INSERT, PROMPT_MUTATE)
improve from 0%?

Updated generate_report.py to handle lv2 calibration data and compare
lv1 vs lv2 results.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
lv1=100%, lv2=0% — both have zero variance. Mixed calibration uses
4 lv1 (PASS) + 3 lv2 (FAIL) for training, giving ~57% seed score.
Evolution can now discover mutations that help solve lv2 tasks
(adding researcher node, improving builder prompt, etc).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…gress

Key findings:
- lv1: Both 4-node and builder-only seeds score 100% (zero variance)
- lv2: Builder-only scores 0% (too hard, all 10 instances FAIL)
- Mixed lv1+lv2 evolution running to find optimal workflow

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Complete calibration results:
- lv1: 100% for both 4-node and builder-only (zero variance)
- lv2: 0% for builder-only (too hard)
- Mixed lv1+lv2: 57% seed score with meaningful variance

Key finding: designer's 3-node variant (researcher+builder+health_checker)
scores identical to builder-only. The builder prompt is the bottleneck,
not the workflow graph structure.

Evolution running with mixed calibration, results pending.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@akashgit akashgit changed the title feat: outer loop v1 phase 4 — CLI, checkpoints, observability, adaptive timeout feat: builder-only seed + mixed lv1/lv2 calibration for outer loop evolution Aug 16, 2026
… events

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Outer Loop v1 Fixes: Leftover Work — Phase 4 + Code Gaps

1 participant