fix: Outer Loop v1 Post-Mortem — All 10 Fixes

[akashgit]

Closes #1272

Changes

Implements all 10 fixes from the outer loop v1 post-mortem across 3 priority phases:

P0 — Evaluation Integrity (critical)

• Fix Validate factory pipeline end-to-end on cloud-gateway #1: Block web search during evaluation — Added --disallowedTools WebSearch,WebFetch to agent invocations and --network none to Docker containers in direct_evaluator.py. Prevents answer leakage (32.67% of SWE-bench solutions involved direct copying).
• Fix Design multi-agent orchestration protocol #2: Simplify fitness to raw pass rate — Replaced broken composite formula (0.6*benchmark + 0.2*hygiene + 0.1*(1-cost) + 0.1*(1-complexity)) with fitness = benchmark_score in evaluator.py. The hygiene/cost/complexity terms were all constant 0.0, adding a fixed 0.185 offset.
• Fix Write SKILL.md v2 workflow #3: Holdout validation every generation — Added audit_generation() to OverfitDetector with per-generation history tracking, early stop after 3 consecutive overfit generations, and overfit_delta field on GenerationSummary.

P1 — Core Differentiation (high)

• Fix Add README documentation #4: Calibrated training/holdout split — New CalibratedSubsetSelector in subset.py that runs a seed workflow, filters to 0.3-0.7 difficulty range, stratifies by repository. Added training_size, holdout_size, difficulty_range to SwarmConfig.
• Fix Rewrite factory's own evals with meaningful metrics #5: Designer workflow prompts — Added populate_prompt() with role-specific templates referencing task files and testbed paths. All DesignerAgent methods now inject benchmark context into generated workflows.
• Fix Fix glob pattern matching in guards.py #6: PROMPT_MUTATE operator — New 7th mutation operator in mutations.py using EvoPrompt-style crossover with frozen segment protection (MUST/MUST NOT/FORBIDDEN/NEVER preserved). Registered at 15% weight in WeightedRandomStrategy.

P2 — Performance & Completeness (medium)

• Fix Fix command parsing in eval script generator #7: Parallel evaluation — evaluate_batch() now uses ThreadPoolExecutor with configurable parallelism (default 4). Added parallelism to SwarmConfig.
• Fix Fix command parsing in eval script generator #8: Clean generation lifecycle — Refactored evolve_generation() into _evaluate_population() → _evaluate_holdout() → _select_and_mutate() → _log_generation(). Added early stop on overfitting.
• Fix Fix glob pattern matching in guards.py #9: Per-instance result tracking — Added instance_results: dict[str, bool] to Individual with per_instance_summary() method. Results extracted from EvalResult.details.
• Fix Rewrite factory evals with meaningful metrics #10: Functional INSERT_NODE prompts — insert_node() now selects roles based on surrounding topology and populates prompts via populate_prompt().

Tests

• 276 tests passing (all tests/test_outer_loop/)
• New test_postmortem_fixes.py with dedicated tests for all 10 fixes plus integration tests
• Existing tests updated for new fitness formula and model changes

Next: Phase 4 — Operational Validation

The code fixes are complete. Phase 4 (tracked in .factory/strategy/current.md) validates the system end-to-end on FeatureBench: calibration run → evolutionary run → learn-iterate cycle.

Files Changed

• factory/outer_loop/models.py — SwarmConfig, GenerationSummary, Individual, MutationType
• factory/outer_loop/evaluator.py — Raw pass rate, parallel evaluate_batch
• factory/outer_loop/direct_evaluator.py — Web search blocking, network isolation
• factory/outer_loop/engine.py — Clean lifecycle, holdout integration, early stop
• factory/outer_loop/overfit.py — audit_generation(), should_early_stop()
• factory/outer_loop/designer.py — populate_prompt(), benchmark context injection
• factory/outer_loop/mutations.py — PROMPT_MUTATE operator, functional INSERT_NODE
• factory/outer_loop/subset.py — CalibratedSubsetSelector
• factory/outer_loop/__init__.py — New exports
• tests/test_outer_loop/test_postmortem_fixes.py — New (65 tests)
• tests/test_outer_loop/test_engine.py — Budget adjustment for holdout evals
• tests/test_outer_loop/test_evaluator.py — Raw pass rate test
• tests/test_outer_loop/test_models.py — MutationType count update

[github-actions]

Sentrux Quality Report

Absolute

Scanning ....
[scan] git ls-files: 642 total, 630 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 630 files, 104 unique dirs, 96 cache misses, 5.1ms
[resolve] 1203 resolved, 1605 unresolved (of 2808 total specs)
[resolve_imports] project_map 5.3ms, suffix_idx 1.2ms, suffix_resolve 19.2ms, total 25.7ms
[build_graphs] 630 files | maps 2.2ms, imports 25.9ms, calls+inherit 7.7ms, total 35.7ms | 1202 import, 9678 call, 11 inherit edges
sentrux check — 3 rules checked

Quality: 4708

✗ [Error] max_cc: 3 function(s) exceed max cyclomatic complexity of 30
    factory/cli/_ceo_helpers.py:_validate_ceo_flags (cc=43)
    factory/cli/_ceo_helpers.py:_execute_ceo (cc=43)
    factory/cli/run.py:cmd_run (cc=32)

✗ 1 violation(s) found

Diff (vs base branch)

Scanning ....
[scan] git ls-files: 642 total, 630 kept, 12 dropped (ext:12, meta:0, big:0)
[build_project_map] 630 files, 104 unique dirs, 96 cache misses, 5.3ms
[resolve] 1203 resolved, 1605 unresolved (of 2808 total specs)
[resolve_imports] project_map 5.4ms, suffix_idx 1.1ms, suffix_resolve 19.5ms, total 26.1ms
[build_graphs] 630 files | maps 1.9ms, imports 26.2ms, calls+inherit 6.3ms, total 34.5ms | 1202 import, 9678 call, 11 inherit edges
sentrux gate — structural regression check

Quality:      4693 -> 4708
Coupling:     0.79 → 0.78
Cycles:       3 → 3
God files:    3 → 3

Distance from Main Sequence: 0.34

✗ DEGRADED
  ✗ Complex functions increased: 56 → 61