Outer Loop v1: Post-Mortem and Fix Plan

[akashgit]

Outer Loop v1: Post-Mortem and Fix Plan

What We Built

The outer loop evolves Factory workflow graphs (DAGs of agents) against benchmarks using evolutionary search. A workflow graph defines which agents run, in what order, with what prompts. The goal: automatically discover workflow topologies that solve coding tasks better than hand-designed ones.

Current Pipeline

┌─────────────────────────────────────────────────────┐
│                    SwarmEngine.run()                 │
│                                                     │
│  1. Seed population (4 candidates):                 │
│     - Slot 0: seed workflow (unchanged)             │
│     - Slot 1: random mutation of seed               │
│     - Slot 2: minimal design (3 nodes)              │
│     - Slot 3: thorough design (10 nodes)            │
│                                                     │
│  2. For each generation:                            │
│     ┌───────────────────────────────────────┐       │
│     │  For each candidate (SEQUENTIAL):     │       │
│     │    For each training instance:        │       │
│     │      1. Extract testbed from Docker   │       │
│     │      2. Run agents on host            │       │
│     │      3. Copy changes into Docker      │       │
│     │      4. Run pytest → pass/fail        │       │
│     │    Score = resolved / total            │       │
│     └───────────────────────────────────────┘       │
│     Select parents (tournament, k=3)                │
│     Mutate parents → offspring                      │
│     Evaluate offspring (same sequential loop)       │
│     Update MAP-Elites archive                       │
│                                                     │
│  3. After all generations:                          │
│     Run holdout audit on best workflow (ONCE)       │
│     Export best workflow                             │
└─────────────────────────────────────────────────────┘

How a Single Evaluation Works

┌──────────────────────────────────────────────────┐
│  DirectFeatureBenchEvaluator._eval_instance()    │
│                                                  │
│  Docker Image ──extract──→ /tmp/testbed/         │
│  (pydantic repo at specific commit,              │
│   implementation scrambled by setup_patch)        │
│                                                  │
│  For each AgentNode in topological order:        │
│    factory agent <role> --task "<prompt>"         │
│      --project /tmp/testbed/ --timeout N         │
│    (runs Claude on the HOST, modifies testbed)   │
│                                                  │
│  Restore test files (reverse test_patch.diff)    │
│                                                  │
│  docker create <image>                           │
│  docker cp testbed changes → container           │
│  docker exec: pip install -e . && pytest         │
│  Exit code 0 → resolved=True                    │
└──────────────────────────────────────────────────┘

Seed Workflow

researcher ──→ builder ──→ health_checker ──→ gate
   (300s)       (7200s)       (600s)

researcher: "Study the codebase and task. Read /tmp/task-instruction.md.
             Explore the repository structure. Write findings."
builder:    "Read task description. Read study output. Implement the
             feature. Run tests. Fix failures. Commit."
health_checker: "Run test suite. Report results."
gate:       "Check if any code was committed."

First Experiment Results

Config: 3 training instances (pydantic lv1), 2 holdout, population 4, budget 20, 4 generations, ~5.5 hours.

All Evaluations

Eval	Gen	Nodes	Type	Inst 1	Inst 2	Inst 3	Train Score
0	0	4	Seed (R→B→HC→G)	✅	✅	✅	1.00
1	0	1	Mutation (1 node)	❌	❌	—	0.00
2	0	3	Designer minimal	❌	❌	❌	0.00
3	0	10	Designer thorough	❌	❌	❌	0.00
4	0	3	Offspring of seed	✅	✅	✅	1.00
5	0	6	Offspring of seed	✅	✅	✅	1.00
6	2	5	Offspring	—	✅	✅	0.67
7	3	3	Holdout audit	✅	❌	—	0.50

Holdout score: 0.485. Training score: 0.785. Overfit delta: 38%. FAILED.

Resource Usage

• 36 total agent invocations (17 researcher, 13 builder, 6 health_checker)
• 20 evaluation budget consumed
• ~5.5 hours wall clock
• 8 full candidate evaluations completed

Critical Analysis

Problem 1: No Swarm — Everything Is Sequential

The plan says "swarm of parallel experiments." The implementation has evaluate_batch(parallelism=1) with a docstring that says "Currently sequential; parallelism is reserved." Every candidate is evaluated one after another. 4 candidates × 3 instances × ~10 min = 2 hours per generation. This makes evolutionary search impractical — real evolution needs parallel evaluation.

Problem 2: Holdout Only Runs Once at the End

The OverfitDetector.audit() is called exactly once, after all generations complete. During evolution, every candidate is scored only on training instances. There is no per-generation holdout tracking. This means:

• We have no idea if candidates generalize during evolution
• A workflow could overfit for 3 generations straight and we'd only discover it at the end

The holdout score should be computed every generation for the best candidate and tracked in GenerationSummary. Holdout is the metric we report — it's the only way to know if evolution is producing real improvement vs memorizing the training set.

Problem 3: Training Set Was Too Easy — No Evolutionary Signal

The seed scored 1.0 on all 3 pydantic lv1 training problems in generation 0. Evolution had no signal to optimize on — you can't beat 100%.

Two separate problems here:

1. Too few instances. 3 is not enough for a meaningful score. Need 10+ to get a stable signal.
2. Too easy. The seed (4-node workflow with researcher+builder+health_checker) solved everything. The seed should be SIMPLE — just a bare builder with no helpers — so evolution can discover that adding a researcher or QA agent actually helps.

Problem 4: Researcher Agent Has Web Search — Answer Leakage

The researcher agent gets WebSearch and WebFetch tools. FeatureBench tasks come from real open-source repos (pydantic, fastapi, etc.). The solutions exist on GitHub, StackOverflow, and blog posts. The researcher can literally google the answer.

The FeatureBench instruction says "You are forbidden to access https://github.com/pydantic/pydantic" but this is text in instruction.md, not an enforced tool constraint. The agent can still find solutions through indirect search results.

Problem 5: Designer Workflows Don't Work

The DesignerAgent.design_minimal() and design_thorough() create workflows with generic prompts that don't reference /tmp/task-instruction.md. The seed works because its prompts are carefully tuned to the FeatureBench environment. The designer workflows are structurally diverse but functionally broken.

All 3 designer/mutation variants scored 0.0 in generation 0. The "diversity" is fake — these candidates can't even read the task description.

Problem 6: Mutations Can Only Simplify, Not Improve

The 6 mutation operators modify graph structure (add/remove nodes, change edges) but don't touch prompts. Since a workflow's effectiveness is ~90% determined by its prompts:

• Removing a node might work (if it was unnecessary) or break the workflow
• Inserting a node adds a do-nothing agent (no prompt = no value)
• The only productive mutations are removing unnecessary nodes

The evolution is just pruning, not improving. The best workflow (3 nodes) is the seed with the health_checker removed — a simplification, not an innovation.

Problem 7: Generation Lifecycle Is Messy

Generation 0 had 6 evaluations (4 initial + 2 offspring). Generation 1 appears empty (starts and immediately transitions to generation 2 at the same timestamp). Generation 2 had 1 evaluation. Generation 3 had 1 evaluation (holdout). The budget didn't drain cleanly across generations.

Problem 8: Fitness Signal Is Broken

The composite score formula is:

fitness = 0.6 * benchmark + 0.2 * hygiene + 0.1 * (1-cost) + 0.1 * (1-complexity)

But hygiene_score is always 0.0 (never computed) and cost_usd is always 0.0 (direct evaluator doesn't track tokens). The multi-metric protection against Goodhart's Law is non-functional. The actual signal is just 0.6 * benchmark + 0.185.

Problem 9: No Per-Instance Tracking

We don't store which instances each candidate solved across generations. This means we can't do instance-level failure analysis, consistency checks, or smart subset rotation.

What Works

Despite the problems, these components are solid and reusable:

• MAP-Elites archive with 4D feature grid
• Novelty filter (hash + graph edit distance)
• 6 structural mutation operators with validation and frozen node protection
• MutationStrategy protocol (pluggable, Level 3 ready)
• HyperparameterRecord per-generation logging
• Workflow serialization round-trip (all 11 builtin workflows)
• Docker evaluation pipeline (extract → agents on host → docker cp → pytest)
• 218 unit tests passing
• SwarmConfig with training/holdout separation

Fix Plan (Priority Order)

Fix 1: Block web search in evaluator

Add --disallowedTools WebSearch,WebFetch to agent invocations in DirectFeatureBenchEvaluator._run_workflow_agents(). Simplest fix, biggest integrity impact. Already partially implemented (commit eb97ec84).

Fix 2: Simplify fitness to raw pass rate

Remove the broken composite formula. Score = resolved / total. Nothing else. Training score drives parent selection within a generation. Holdout score is the reported result. No hygiene, no cost normalization, no complexity penalty — these were all zeros anyway.

Training score is useful signal for directing the search (which mutations are working, which parents to select from). It's not noise. But holdout score is what we report as the outcome — it's the only way to know if the evolved workflow actually generalizes.

Fix 3: Holdout every generation

Move holdout evaluation into evolve_generation(). After evaluating the population on training instances, evaluate the best candidate on holdout. Track holdout_score in GenerationSummary. This gives per-generation visibility into generalization. Already partially implemented (commit eb97ec84).

Important: holdout score is for REPORTING only. Do NOT use it for parent selection — that would let evolution overfit to holdout too, just more slowly. Training drives search, holdout measures generalization.

Fix 4: Calibrate difficulty — 10 training, 3-5 holdout

Before running evolution:

1. Run the SIMPLEST possible workflow (just a bare builder, 1 node, no researcher) on 20-30 FeatureBench problems
2. Pick 10 where the simple builder scores 0.3-0.7 (partial success) — these are problems where a better workflow could help
3. Pick 3-5 additional problems at similar difficulty for holdout
4. Discard problems where the builder scores 0.0 (too hard) or 1.0 (too easy)

The seed for evolution should be this simple builder (1 node). Evolution should discover that adding a researcher, code reviewer, or QA agent improves the score. Starting with a 4-node seed that already aces everything leaves no room for improvement.

Fix 5: Fix designer prompts

All workflows (designed, mutated, or seeded) must include the critical benchmark-specific prompt content: "Read /tmp/task-instruction.md", implementation rules, commit rules. The topology can vary but the essential instructions must be present. Otherwise designed workflows are structurally diverse but functionally dead.

Fix 6: Add prompt mutation operator (simple version)

Add PROMPT_MUTATE to the mutation type enum. When selected:

1. Pick a random AgentNode in the workflow
2. The Designer Agent reads: the current prompt, the benchmark spec, and failure telemetry from the last evaluation
3. The Designer Agent proposes a revised prompt (complete replacement, not a diff)
4. This is a single LLM call per mutation

This gives evolution two levers: topology (which agents, what order) AND prompts (what instructions each agent gets). Without this, evolution can only prune nodes — it can't make agents better at their jobs.

This is a simple version, not full bi-level optimization. The full version (outer loop evolves topology, inner SkillOpt loop optimizes prompts within each topology) is v2. For now, prompt mutations are just another mutation operator alongside the structural ones.

Fix 7: Parallel evaluation

Use concurrent.futures.ProcessPoolExecutor in evaluate_batch(). Each candidate gets its own temp directory and Docker container. Target: 4 concurrent evaluations on the remote-claude machine (12 CPU, 64GB RAM via Colima).

Fix 8: Clean generation lifecycle

Fixed structure per generation:

1. Evaluate current population on training instances
2. Evaluate best on holdout instances (for reporting)
3. Select parents from archive (using training scores)
4. Generate offspring via mutation
5. Log GenerationSummary with both training and holdout scores

No skipping, no merging, no variable sizes.

Fix 9: Per-instance tracking

Store instance-level results (which instances each candidate solved/failed) in the archive. Enables failure analysis and consistency checks.

Fix 10: Verify insert_node creates functional agents

The insert_node mutation operator needs to create AgentNodes with functional prompts that include the benchmark-specific patterns (task file path, implementation rules), not empty placeholders. If the operator just inserts a generic node with no prompt, the mutation is structurally valid but functionally useless.

Learnings

1. Training score is useful signal, holdout is the only reported result. Training drives the search — which mutations work, which parents to select. Holdout measures whether the result generalizes. Both matter, for different purposes.
2. Topology-only evolution without prompt evolution is pruning, not improving. The 10-node workflow scored 0 because its prompts were wrong, not because 10 nodes is a bad topology.
3. Web search = answer leakage on benchmark tasks. Must be blocked for fair evaluation.
4. Calibration before evolution is mandatory. Start with a simple seed (bare builder). Use problems where the simple seed scores 0.3-0.7 so evolution has gradient to climb.
5. Sequential evaluation makes evolution impractical. A real run takes hours. Parallelism is not optional.
6. Designer-created workflows must be functional, not just diverse. Structural diversity without working prompts is zero signal.

Files

• factory/outer_loop/engine.py — SwarmEngine (412 lines)
• factory/outer_loop/evaluator.py — SwarmEvaluator + FitnessCache (139 lines)
• factory/outer_loop/direct_evaluator.py — FeatureBench evaluator (441 lines)
• factory/outer_loop/mutations.py — 6 operators + MutationStrategy (557 lines)
• factory/outer_loop/population.py — Population + MAPElitesArchive (203 lines)
• factory/outer_loop/similarity.py — hash, GED, NoveltyFilter (134 lines)
• factory/outer_loop/overfit.py — OverfitDetector (78 lines)
• factory/outer_loop/designer.py — DesignerAgent dual mode (344 lines)
• factory/outer_loop/models.py — Pydantic models (184 lines)
• factory/outer_loop/run_evolution.py — CLI runner (133 lines)
• PR: feat: outer loop Phase 1 — population, mutations, similarity, serialization #1258 targeting feat/outer-loop-evolution