feat: add smoke test harness for core factory modes - #4
Draft
osilkin98 wants to merge 369 commits into
Draft
Conversation
…ce persistence (akashgit#963, akashgit#964, akashgit#965) (akashgit#966) * fix: add fallback handler for contributed workflow modes in CEO task builder _build_ceo_task() had explicit elif blocks for build, discover, meta, research, and create modes but no fallback for contributed/custom modes like swebench or legacybench. Without instructions, the CEO would waste time figuring out what to do instead of reading the mode's SKILL.md. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove auto_merge FnNode from swebench workflow (akashgit#963) Harbor's recovery chain already handles merging changes to main, making the auto_merge node redundant. Pipeline is now 3 nodes (study → builder → gate_verify) with RELOOP on test failure. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add terminal flag to Workflow model to prevent post-completion chaining (akashgit#964) Workflows marked terminal=True cause _chain_modes() to return immediately after the mode completes, preventing unwanted auto-chaining into discover/ improve modes. Sets terminal=True on the swebench workflow. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: persist trace_id.txt to Harbor /logs/ so benchmark scripts can find it (akashgit#965) Two bugs prevented trace_id extraction after benchmark runs: 1. trace_id.txt was written inside the container's project dir but never copied to /logs/ (the only path Harbor exports to JOBS_DIR) 2. Benchmark cleanup functions deleted JOBS_DIR before searching it for trace_id.txt, so even with the file present it would never be found Fix: copy .factory/trace_id.txt to /logs/.factory/ in both Harbor agent classes after factory ceo exits, and reorder cleanup() in all 5 benchmark scripts to extract trace_id before rm -rf. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…hgit#968) Without auto_merge, the factory makes the correct fix but it stays on the worktree branch — which the factory deletes on exit. Harbor's recovery chain can't find the changes, so the benchmark fails despite a correct solution. Uses `git branch -f $BASE HEAD` instead of `git checkout && git merge` to update main's ref without checking it out — works inside worktrees where checkout would fail. Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…-gate # Conflicts: # factory/workflow/definitions.py # tests/test_workflow_definitions.py # tests/test_workflow_qa.py
The old monolithic qa node (writing qa-latest.md) was replaced by the deep-QA pipeline. Update gate_doc_freshness to read adversarial-qa.md from the adversarial_tester node, which is the last step in the deep-QA chain. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…nd doc-update workflows - Extract the duplicated gate_prompt string from 4 workflow definitions into a module-level DOC_FRESHNESS_GATE_PROMPT constant - Add doc_generate_workflow(): scan_project -> gate -> generate_docs -> gate -> validate_docs -> gate (standalone, no trigger) - Add doc_update_workflow(): diff_scope -> patch_docs -> gate -> revalidate -> gate (standalone, no trigger) - Register both as 'doc-generate' and 'doc-update' in register_all() - Add comprehensive tests for both workflows and constant usage Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ce_id (akashgit#970) - Use `factory workflow run swebench .` instead of `factory ceo --mode swebench` (CEO paraphrases shell commands; executor runs them verbatim) - Fix auto_merge: `git update-ref` + `git-common-dir` for correct branch detection + copy changed files to parent working tree - Fix trace_id: copy to /logs/agent/ (Harbor-exported path), search by -name Verified: 5/5 local Harbor runs pass with trace_id captured. Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ess-gate feat: add gate_doc_freshness GateNode to 5 factory workflows
Change tension from 0.2 to 0 in all Chart.js dataset configs so charts render linear segments instead of bezier curves. Closes akashgit#971 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Inject trace summary, analysis, and Langfuse URL into results.jsonl during CI, and render them in the benchmarks dashboard with expandable detail rows and clickable trace links. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- HTML-escape trace_summary in renderPerBenchmarkTable() before injecting as innerHTML, consistent with trace_analysis escaping - Use with-statement for file reads in benchmark.yml to ensure proper file handle cleanup Closes akashgit#973 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: use straight line segments in benchmark charts
Change tension from 0.2 to 0 in all Chart.js dataset configs so charts render linear segments instead of bezier curves. Closes akashgit#971 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…#954) FactoryCeo was hardcoded to run the swebench workflow for all benchmarks. Extract _get_factory_command() as an override point, restore generic factory ceo for featurebench/terminalbench, and add SwebenchFactoryCeo and LegacybenchFactoryCeo subclasses with their respective workflows. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
feat: display trace analysis on benchmarks dashboard
Seed .factory/ with minimal config.json and eval_profile.json so state detection yields has_factory → improve mode, then use --focus with the instruction text instead of --mode build --prompt. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The seeded eval_profile.json used "reviewed" but the EvalProfile model and state detection check "human_reviewed". This caused state detection to return EVALS_PENDING_REVIEW instead of HAS_FACTORY, breaking --focus for featurebench/terminalbench runs. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…-split fix: split FactoryCeo into benchmark-specific agent classes
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…k scope Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Harbor containers installed factory from main, so new workflows
(terminalbench, featurebench) weren't available until merged.
Now passes FACTORY_GIT_REF=${{ github.sha }} through the benchmark
workflow → run scripts → Docker container, and the install step
uses it to pin the factory version to the commit being tested.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
${REF} was inside single quotes so the shell didn't expand it.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
_parse_fn_verdict checked for "fail" anywhere in the output text, which matched "no failure signals detected" and halted the workflow even on a pass verdict. Now checks the first line's prefix instead: "pass:" → proceed, "fail:"/"revert:" → halt, "reloop:" → reloop. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…-split feat: add terminalbench and featurebench deterministic workflows
…kflow dependency Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…-split fix: rewrite legacybench as lean pipeline — remove broken improve_workflow dependency
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…-split feat: add programbench discovery-first workflow
…git#982) Without this, Claude Code uses its default max_turns which can be as low as 3 in fresh CI installs, causing agents to be killed before finishing their work. Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: integrate DevOps Gym as contributed benchmark (akashgit#1152) Add DevOps Gym build/configuration benchmark following the proven contributed benchmark pattern (legacybench, swebench, etc.). - 4-node pipeline: study -> solver -> gate_verify -> auto_merge - Study node detects build systems (Maven, Gradle, Go, Make, Docker) - Gate verify attempts build with detected build tool - Harbor agent class DevOpsGymFactoryCeo runs workflow deterministically - Registered in definitions.py, config.sh, run.sh, benchmark.yml - 26 structural tests covering graph, trigger, registration, and meta * fix: update workflow count assertion for devopsgym The devopsgym contributed workflow adds a 34th workflow to the registry. Update test_register_all_count assertion from 33 to 34. --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com>
feat: add CLI plugin architecture via entry points
…ssertion Two bugs prevented plugin modes from working: 1. _validate_ceo_flags checked mode against the static CEO_MODES list instead of get_all_ceo_modes(), rejecting plugin-registered modes (e.g. attack, buildroot) as "unknown mode". 2. cmd_ceo asserted raw_path is not None before running pre-hooks. Plugin modes that create their work directory via add_ceo_pre_hook crashed on the assert because no positional path was provided — the pre-hook that would supply it never got to run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
mypy caught that raw_path can be None when pre-hooks are registered. Add a truthiness check before passing it to Path() in the --refine validation branch. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…tion fix: plugin modes rejected and pre-hooks crash on missing path
…ktrees (akashgit#1235) * fix: replace directory symlink with selective symlinks in CEO run worktrees Closes akashgit#1234 The single .factory/ directory symlink caused context contamination (stale strategy/review files from prior cycles) and race conditions (concurrent CEO runs writing to the same mutable state). Replace with a real .factory/ directory that selectively symlinks shared append-only state (config, results.tsv, experiments, etc.) and creates fresh per-cycle directories (strategy, reviews, state). Backlog.md is copied in at creation and synced back at teardown. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update worktree tests to assert selective symlink layout Tests were still checking for the old directory-symlink behavior where .factory was a single symlink. Updated to assert the new layout: .factory is a real directory with selective symlinks inside (config.json) and per-cycle real directories (strategy/). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: add adversarial_state.json and performance_report.json to shared symlinks Both files are cross-cycle project-wide state that was shared via the old directory symlink but missing from _SHARED_SYMLINK_ENTRIES. Without them, adversarial convergence tracking resets each cycle and ACE loses qualitative signals from performance reports in worktrees. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…1241) The slow update was injecting guidance into self.current_skill (rendered SKILL.md) but never into the YAML annotations prompt slots. Since the adapter serializes YAML for Harbor, the guidance never reached the agent inside the container. Now injects into the primary prompt slot (largest prompt slot) and writes YAML annotations, so the guidance flows through: prompt_slot → YAML → FACTORY_WORKFLOW_YAML_B64 → container → agent. Also uses _serialize_yaml() for the prev/curr rollouts so the comparison runs use the correct mechanism. Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add study mode with graph-powered code exploration Add a standalone Study Mode workflow and integrate graph exploration into the design workflow's study subgraph. Study subgraph (shared by design and study workflows): graph_update → study → graph_explorer → concat_study - graph_update: runs `factory graph update` to build/refresh graph.json - study: runs `factory study` producing observations.md - graph_explorer: researcher agent that reads observations, queries the code graph via `factory graph query/explain/path`, and writes graph-context.md with structural findings - concat_study: merges observations.md + graph-context.md into study-combined.md for downstream consumers Design workflow integration: - Existing projects route through the study subgraph before research - Researchers and strategist read study-combined.md for project context - join_research is now a pure sync barrier (no concatenation) - gate_research and strategist read individual research files directly CLI additions: - `factory graph query/explain/path` — agent-accessible graph tools - `factory ceo --mode study` / `factory run --mode study` - Study mode registered across all integration points Also updates researcher agent prompt to read study-combined.md instead of re-running factory study redundantly when study has already run upstream in the workflow. * fix: update plan workflow and registry count tests Plan workflow inherits the study subgraph from design_workflow, so node count goes from 15 to 18 and edge list includes the subgraph edges. Registry count goes from 34 to 35 with the new study workflow.
Plugin modes registered via add_modes() failed Pydantic validation when headless mode created a CycleState, since the Literal type only accepted hardcoded built-in mode names. Closes akashgit#1262 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Plugin modes registered via add_modes() failed Pydantic validation when headless mode created a CycleState, since the Literal type only accepted hardcoded built-in mode names. Closes akashgit#1262 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…it#1260) The hardcoded choices list in argparse rejected plugin-registered roles before the agent runner could execute. Move validation to cmd_agent() where it checks both BUILTIN_AGENT_ROLES and plugin-registered roles via get_registry().agent_roles. Add add_agent_roles() to PluginRegistry following the same collision-guard pattern as add_modes(). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
After the existing symlink/copy logic and per-cycle fresh dirs, iterate over remaining subdirectories in source .factory/ and copy any not already handled. Skips entries already present (symlinked, copied, or created fresh) to avoid duplication. Closes akashgit#1263 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ole to str Closes akashgit#1264 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…-bugs fix: plugin integration — 4 bugs blocking plugin modes and agents
…egist plan reading Fixes akashgit#1204, akashgit#1205, akashgit#1206. Converts the sequential deep-QA pipeline (health_checker → code_reviewer → gate_review → adversarial_tester) to parallel execution via fork/join: fork_qa → [health_checker, code_reviewer, adversarial_tester] → join_qa Key changes: 1. Worktree anchoring (akashgit#1204): All 3 QA agent prompts now include a "Working Directory Constraint" preventing agents from navigating to parent directories or other worktrees. 2. Strategist plan reading (akashgit#1205): adversarial_tester.md now has an explicit Step 0 that reads .factory/strategy/current.md to derive testing scope from hypothesis deliverables. 3. Parallel execution (akashgit#1206): _deep_qa_subgraph() now returns fork/join nodes instead of sequential edges. All workflows (build, improve, research, refine, create, deep-qa, parallel-improve) updated. 4. Graph validator: added implicit edges for ForkNode targets and JoinNode sources in reachability analysis. 5. parallel-improve: fixed SubgraphForkNode renaming to also rename ForkNode.targets and JoinNode.sources when namespacing. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_gate_qa_topology: check for fork_qa instead of sequential health_checker/code_reviewer/gate_review/adversarial_tester nodes - test_e2e_gate_before_improve: check builder < fork_qa in topo order instead of builder < health_checker (fork targets have no ordering guarantee relative to other nodes) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Fixes 3 issues from factory CEO review:
1. _topological_sort (skill_export.py) now adds implicit edges for
ForkNode targets and JoinNode sources, so fork children sort after
the fork node. This fixes loop context topology rendering and skill
export ordering.
2. Removed stale sequential prerequisites from all 3 QA prompts:
- adversarial_tester.md: removed "health check must have passed"
and "code review must have found no critical issues"
- code_reviewer.md: removed "health check must have passed"
- health_checker.md: removed "proceed to code review" gate language
All now state they run in parallel with the other agents.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
feat: parallelize deep-QA pipeline + worktree anchoring + strategist plan reading
… review file
When multiple AgentNodes share the same role, the generic
reviews/{role}-latest.md file from the first node caused all subsequent
same-role nodes to be auto-skipped. Fix by promoting the declared writes
check above the generic review file check, and returning None after the
writes check to prevent fallthrough.
Closes akashgit#1277
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ode-literal fix: widen CycleState.mode from Literal to str for plugin modes
fix: tool engine auto-skips agent nodes when multiple nodes share the same role
…#1279) (akashgit#1283) Replace check-then-act (_session_exists → branch) with EAFP pattern: try new-session first, fall back to new-window on non-zero returncode. When parallel agents race on session creation, only one new-session wins; the others now fall back to new-window instead of failing. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…shgit#1233) * feat: add glaude runner (Claude Code via GLM-5.2 LiteLLM proxy) Closes akashgit#1229 - Create factory/runners/glaude.py mirroring ClaudeRunner with binary 'glaude', empty required_env_vars (auth baked into wrapper), and full capability flags (telemetry, background, session naming, model override) - Register GlaudeRunner in factory/runners/__init__.py - Add FACTORY_GLAUDE_DRY_RUN support for testing without tokens - Add 41 tests covering runner selection, dry-run, metadata, command building, headless invocation, interactive mode, and temp file cleanup - Document Glaude specifics in CLAUDE.md runner section Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * Revert "feat: add glaude runner (Claude Code via GLM-5.2 LiteLLM proxy)" This reverts commit 16fd72a. * feat: add env overlay support to config.toml credential profiles Profiles now override existing env vars (not setdefault), support unsetting vars via [credentials.*.unset].vars list, and reject protected system vars (PATH, HOME, etc.). Enables any endpoint variant (e.g. glaude/LiteLLM proxy) via config alone — no new runner class needed. Closes akashgit#1229 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: replace glaude-specific references with generic custom endpoint examples Rename all glaude/GLM-5.2 references to generic placeholder names (litellm-proxy, custom) so the env overlay feature is presented as a general-purpose mechanism for configuring custom model endpoints via profiles, not tied to any specific deployment. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: update load_config docstring to reflect override semantics The docstring still described the old setdefault behavior. Updated to document that profiles override existing env vars, support unsetting vars, and protect PATH/HOME/etc. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * harden config.toml credential profiles: expand protected vars, validate unset types, add override warnings - Expand _PROTECTED_VARS to block code execution vectors (LD_PRELOAD, DYLD_INSERT_LIBRARIES), language path injection (PYTHONPATH, GOPATH, CLASSPATH, NODE_PATH), shell parsing (IFS), and factory internals (FACTORY_TRACE_ID, FACTORY_PARENT_SPAN_ID) - Validate unset.vars is a list, raising ValueError on string/other types - Log structured warning when profile overrides existing env var (no values logged) - Mask sensitive values in nested sub-table rendering in show_config - Check config file permissions and warn if group/other readable - Add 8 new tests covering all hardening behaviors - Document expanded protections in CLAUDE.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: Phase 1 — wire InnerLoop into SwarmEngine with partial credit scoring
Cherry-pick v1 outer_loop foundation from feat/outer-loop-phase1-foundation,
restructure for InnerLoop-based evaluation. Key changes:
- Add Workflow.to_dict()/from_dict() serialization methods
- Create FeatureBenchEvaluator with partial credit scoring (pytest-json-report)
- Create FeatureBenchInnerLoop wrapping InnerLoop.step() for CycleRecord exhaust
- Add CycleRecordCache for content-addressable eval caching
- Modify SwarmEvaluator to support both legacy EvaluatorFn and InnerLoop
- Update DirectFeatureBenchEvaluator for partial credit (float) scoring
- Add parsimony pressure: fitness = score - 0.01 * num_nodes
182 tests passing (excluding CLI tests pending Phase 4).
* feat: Phase 2 — ephemeral mode registry for candidates-as-modes
Create EphemeralModeRegistry with register/cleanup/promote lifecycle:
- Content-addressable storage with hash verification
- evolve-gen{N}-{id[:8]} naming — never collides with main registry
- Context manager protocol for guaranteed cleanup
- Wire into SwarmEngine.seed() and evolve_generation()
- 11 tests passing
* feat: Phase 3 — contrastive reflection agent with exhaust analysis
Create OuterLoopReflector with two-stage contrastive reflection:
- Compare top-K vs bottom-K CycleRecords to find structural differences
- Extract failure/success patterns from agent steps and experiment results
- Generate informed mutation suggestions based on role and topology diffs
- Save reflection reports to .factory/outer_loop/reflections/
- Wire into SwarmEngine between evaluate and evolve phases
- Add MAX_NODES=30 bloat prevention in mutation operators
- Create reflector.md and evolver.md agent prompts
- Create reflect.md prompt template
- 7 reflector tests + 200 total passing
* feat: Phase 4 — outer-loop as registered mode with CLI subcommands
- Create workflow graph: seed → evaluate → reflect → evolve → gate_converge
- Add CLI subcommands: calibrate, evaluate, reflect, evolve, status, promote
- Register outer-loop in CEO_MODES and main CLI dispatch
- Add outer-loop to Self-Evolution command group
- 17 CLI tests + 217 total passing
* feat: Phase 5 — convergence detection with cost logging and diversity monitoring
- Add configurable convergence criteria to SwarmConfig: plateau_window,
plateau_threshold, diversity_floor, early_stop_unchanged
- Implement 3 convergence detectors: fitness plateau (< 1% improvement
over N generations), diversity collapse (< 20% of initial), early stop
(top-3 unchanged for N generations)
- Add event logging to .factory/outer_loop/events.jsonl (diversity, fitness)
- Add cost logging to .factory/outer_loop/costs.jsonl (per individual)
- BudgetTracker is logging/tracking only — never limits execution
- 217 tests passing
* fix: lint — remove unused imports and fix f-string
* fix: wire reflection→mutations, implement CLI stubs, add PROMPT_MUTATE, cleanup modes
6 fixes from deep QA pipeline review:
1. Wire reflection to mutations (CRITICAL): pass ReflectionReport from
engine.py to apply_random_mutation; 70% guided / 30% random selection
via WeightedRandomStrategy.select_guided_operator()
2. Implement 3 stub CLI subcommands: evaluate, reflect, evolve now call
real SwarmEngine methods (load config, evaluate population, run
reflection, produce offspring)
3. Add PROMPT_MUTATE operator: 7th mutation type that modifies AgentNode
prompt_template with strategy variants or reflection-derived hints
4. Wire cleanup_generation(): called after offspring evaluation in
evolve_generation() to remove non-surviving ephemeral mode files
5. Fix lint + type errors: Sequence instead of list for covariant type
hints in reflector.py; type-safe dict access in direct_evaluator.py;
7 unused imports removed via ruff --fix
6. Add contributed workflow compliance: README.md and test_workflow.py
for factory/workflow/contributed/outer_loop/
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: wire population seeding into outer-loop calibrate command
The calibrate command only initialized the filesystem and wrote config
but never created the initial population of workflow variants. This
caused 'factory outer-loop evaluate' to fail with 'no ephemeral modes
found' because no modes were registered.
Now calibrate loads the base FeatureBench workflow, creates a
SwarmEngine with EphemeralModeRegistry, calls engine.seed() to
populate slots (unmodified seed + mutations + designer variants),
saves the population to checkpoint, and prints the created modes.
Closes akashgit#1282
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: register designer variants as ephemeral modes and wire inner_loop_factory in CLI
Bug 1: _add_designer_variants() added individuals to the population but
never called mode_registry.register(), so only the base seed appeared as
an ephemeral mode. Now registers each designer individual identically to
the seed and mutation slots.
Bug 2: All CLI commands (calibrate, evaluate, reflect) created
SwarmEvaluator without inner_loop_factory, leaving it None. This caused
evaluation to skip InnerLoop.step() entirely and return dummy score=0.0.
Added _make_inner_loop_factory() helper that bridges the registry into
SwarmEvaluator so FeatureBenchInnerLoop is actually invoked.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: write workflow wrappers so ephemeral modes are discoverable by sub-CEO
EphemeralModeRegistry.register() now writes a thin .py wrapper to
.factory/workflows/{mode_name}.py alongside the JSON in outer_loop/modes/.
This makes ephemeral modes discoverable by WorkflowRegistry.discover()
when a sub-CEO process runs 'factory ceo --mode <name>'.
cleanup_generation() and cleanup_all() remove both the JSON and wrapper.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: export meta and workflow from outer_loop contributed package
The __init__.py was empty, causing ImportError in test_workflow.py
which imports from factory.workflow.contributed.outer_loop directly.
Follows the same pattern as other contributed workflows (featurebench).
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add explicit graph detection and project path to Researcher graph commands
The Researcher agent's graph_explorer prompt had two bugs:
1. Ambiguous detection: "if graph.json exists" without specifying WHERE
or HOW to check, causing the agent to report "no graph.json" even
when graph.json existed at the project root
2. Missing path argument: `factory graph query "<q>" --depth 2` omitted
the required `<path>` positional arg (CLI expects `factory graph query
<path> <question> --depth <n>`)
Fix: Add explicit `factory graph status .` detection step and include `.`
(CWD = project root, set by the agent runner) as the path argument in all
graph CLI commands. Also fix the tracked `skills/study/SKILL.md` skill.
Closes akashgit#1256
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: resolve 4 outer loop E2E validation issues
Issue 1 — Set designer_count=0 for featurebench and use a 1-node
builder-only seed workflow instead of the full 4-node pipeline.
Lowered NoveltyFilter min_edit_distance for small seeds so mutations
produce novel variants.
Issue 2 — Added offset-based artifact isolation to CycleAnalyzer and
InnerLoop so each sub-CEO's events/results are scoped by snapshot
offsets, preventing cross-contamination of experiment data.
Issue 3 — Cost attribution automatically scoped by the offset approach
from Issue 2.
Issue 4 — Added --project-dir override to calibrate and evaluate CLI
subcommands for benchmark instance isolation.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: make Researcher graph explorer use explicit paths and smoke check
The graph_explorer prompt told the Researcher to run `factory graph status .`
but Sonnet often skipped that step and checked for graph.json in the wrong
location (e.g. .factory/ instead of the project root). Three changes:
1. Add a `test -f graph.json` smoke check with explicit GRAPH_EXISTS/NO_GRAPH
output so the agent has a concrete, unambiguous detection step
2. Clarify that graph.json lives at the PROJECT ROOT, NOT inside .factory/
3. Replace `.` with `"$(pwd)"` in all graph CLI commands for explicit path
resolution, matching the pattern already used in skills/study/SKILL.md
Closes akashgit#1256
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: run sub-CEO in headless mode for outer loop evaluate
InnerLoop.step() subprocess was missing --headless, causing the sub-CEO
to take the interactive path (runner.interactive_run) which doesn't
reliably execute workflows as a nested subprocess. Adding --headless
routes through run_ceo_with_completion_guard → invoke_agent for proper
one-shot agent execution and event emission.
CycleAnalyzer offset mechanism verified correct — the 0 steps/cost was
a consequence of sub-CEOs not running, not an offset bug.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use ls+pwd smoke check for graph.json detection in Researcher prompt
Replace `test -f graph.json && echo GRAPH_EXISTS` with explicit `pwd` + `ls -la`
in a fenced code block. This gives the agent concrete, visible output to reason
about rather than a boolean flag — easier for Sonnet to follow reliably.
Closes akashgit#1256
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: InnerLoop writes cycle_summary.json for reliable outer loop scoring
The CycleAnalyzer approach of parsing shared .factory/ artifacts is
broken for the outer loop because the sub-CEO writes to the same
.factory/ and doesn't produce experiment records. Instead, InnerLoop
now writes a structured cycle_summary.json after each step() with
observable outcomes (agents spawned/succeeded/failed, builder commits,
subprocess exit code) and a 0.0-1.0 score. The evaluator reads this
file as the primary score source, falling back to CycleRecord.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: inject resolved project path into Researcher graph explorer prompt
The Researcher agent fell back to grep despite graph.json being present
because `$(pwd)` in the prompt was ambiguous — the agent had to run a
shell command to discover the path, and Sonnet frequently reported "no
graph.json" without actually running the check.
Three changes fix this:
1. `_GRAPH_EXPLORER_PROMPT` now uses `{project_path}` template variable
instead of `$(pwd)`, so the actual path is baked into the prompt text
2. `executor._run_agent()` substitutes `{project_path}` with the
resolved project path before passing to the agent
3. `skill_export._agent_node_to_md()` maps `{project_path}` to
`$PROJECT_PATH` for the interactive SKILL.md path
Closes akashgit#1256
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: mirror ephemeral mode wrappers to target project dir for sub-CEO resolution
When `factory outer-loop evaluate --project-dir` targets a different project
(e.g. a FeatureBench instance), the sub-CEO couldn't resolve ephemeral modes
because wrappers were only written to the outer loop project's .factory/.
EphemeralModeRegistry now accepts a target_dir parameter and mirrors mode JSON
and workflow wrapper files to the target directory so the sub-CEO finds them.
Closes akashgit#1256
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: add target_project to SwarmConfig so sub-CEOs evaluate on FeatureBench instance
- Add target_project field to SwarmConfig (defaults to empty string)
- _cmd_calibrate persists --project-dir into config.target_project
- _cmd_evaluate falls back to config.target_project when --project-dir not passed
- Add tests for CLI parsing, model round-trip, and evaluate fallback behavior
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: remove nested triple backticks from graph_explorer prompt
The _GRAPH_EXPLORER_PROMPT contained fenced code blocks (```) for the
smoke check example. When skill_export embedded this prompt inside a
bash code fence in SKILL.md, the nested ``` prematurely closed the
outer code block. The CEO then saw a truncated command and passed an
incomplete task to the Researcher, which never received the graph
detection instructions and always fell back to grep.
Replace the fenced code block with inline code formatting so the
prompt can safely nest inside SKILL.md bash blocks.
Closes akashgit#1256
* fix: use pytest pass rate for scoring and worktree isolation per candidate
Issue 1: InnerLoop._write_cycle_summary now runs the configured
test_command after the sub-CEO completes and uses the pytest pass
rate (passed/total) as the primary score. The old heuristic is
kept as metadata (heuristic_score) but no longer drives scoring.
Issue 2: SwarmEvaluator._evaluate_via_inner_loop creates an isolated
git worktree per candidate so each evaluation starts from a clean
instance state. Worktrees are cleaned up after scoring. evaluate_batch
supports parallel evaluation via ThreadPoolExecutor.
Changes:
- SwarmConfig: add test_command field
- InnerLoop: add test_command param, _run_test_command method
- FeatureBenchInnerLoop: forward test_command to InnerLoop
- SwarmEvaluator: worktree create/cleanup, parallel evaluate_batch
- CLI: --test-command arg on calibrate subcommand
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: outer loop CLI bugs — reflect target, generation filter, eval dedup
Bug 1: _cmd_reflect now uses config.target_project instead of
project_path when evaluating workflows, matching _cmd_evaluate behavior.
Bug 2: _cmd_evaluate filters modes to the specified generation prefix
(evolve-gen{N}-*) and excludes eval copies (evolve-gen{N}-eval-*).
Bug 3: _make_inner_loop_factory checks if an eval mode already exists
before registering, preventing duplicates across CLI re-invocations.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: register outer-loop workflow in builtin registry
The outer-loop contributed workflow was not imported in
definitions.py's _get_builtin_registry(), so the WorkflowRegistry
never discovered it. This caused SKILL.md not found errors when
creating worktrees (the skill cache only includes registered
workflows). Adding the import makes outer-loop a first-class
builtin, fixing both the naming mismatch and the skill cache issue.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add 20 edge-case tests and document outer loop subsystem in CLAUDE.md
H2: Add 8 test coverage gaps (20 tests in test_coverage_gaps.py) covering
graph fallback propagation, worktree cleanup with locked files, eval dedup
logic, disk-full worktree creation, budget exhaustion partial results,
all-identical convergence detection, reflector empty history, and mode
registry hash collisions. Total outer loop tests: 262 (up from 242).
H3: Update CLAUDE.md with Layer 2b (outer loop architecture, key modules,
MAP-Elites pipeline, single-builder E2E finding), outer loop models
documentation, CLI subcommands (calibrate/evaluate/reflect/evolve/status/
promote), and .factory/outer_loop/ directory layout.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: heuristic double-count, outer_loop dir naming, CLAUDE.md paths
- I1: Replace duplicate returncode==0 signal with experiments>0 in
heuristic scoring (inner_loop.py) — each signal now contributes
exactly 0.2 with no double-counting
- I2: Standardize .factory/outer-loop/ → .factory/outer_loop/ in
filesystem.py to match engine.py/cli — fixes status command missing
trajectory.jsonl
- I2b: Fix CLAUDE.md checkpoint.json → state.json to match actual
filename
- I4: Add TestHeuristicScoreWeights (7 tests) verifying each signal
weight and no double-counting
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add cache persistence, stale mode pruning, and disk space pre-check for outer loop
Implements H1 (operational hardening) from the strategy:
1. CycleRecordCache now persists to .factory/outer_loop/eval_cache.jsonl
(append-only JSONL with dedup). SwarmEvaluator loads on init and
exposes checkpoint_cache() for generation-level saves.
2. EphemeralModeRegistry.prune_stale_modes(older_than_hours=24) removes
old mode JSONs and workflow wrappers to prevent unbounded accumulation.
3. _check_disk_space() in the CLI requires population_size*0.2+10 GB free
before calibrate/evolve, failing fast with a clear message.
14 new tests cover all three features.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: outer loop convergence gate, structural hash dedup, best_score tracking
- Add convergence gate to outer-loop SKILL.md: gate_converge now has explicit
PROCEED → promote and RELOOP → evaluate paths so the CEO loops generations
instead of falling into default improve mode
- Fix skill_export _gate_to_checkpoint to show RELOOP (not HALT) when RELOOP
edges exist without HALT edges
- Remove workflow.name from structural_hash so structurally identical workflows
with different names share the same fitness cache entry
- Update _cmd_evaluate to track best_score across generations in state.json
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: stop eval-copy mode accumulation and redundant re-evaluation in outer loop
_make_inner_loop_factory now looks up existing modes by structural hash
instead of creating new eval-copy modes on every evaluation call. _cmd_evaluate
persists cycle_summary.json for each mode, and _cmd_reflect reads those cached
results instead of re-evaluating all candidates sequentially.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore: update uv.lock after outer loop bug fixes
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: mirror ephemeral modes to target project in calibrate and evolve commands
Both _cmd_calibrate and _cmd_evolve created EphemeralModeRegistry
without target_dir, so gen0 and gen2+ modes were never mirrored to
the FeatureBench instance. Sub-CEOs on the target couldn't resolve
these modes, scoring 0.0. Now all CLI commands that register modes
pass config.target_project to the registry.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: use CWD-relative path in graph_explorer smoke check
The Researcher agent's graph_explorer prompt used {project_path}/graph.json
for the smoke check, which becomes $PROJECT_PATH/graph.json in the SKILL.md.
Since $PROJECT_PATH is not set as an env variable in the Researcher's shell,
the check always fails and the agent falls back to grep-based exploration
even when graph.json exists at the project root.
Changed the smoke check to use `test -f graph.json` (relative path) since
the agent's CWD is already set to the project root via --project. Factory
graph CLI commands retain {project_path} since they're resolved by the CEO.
Closes akashgit#1256
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: propagate InnerLoop test results through evaluator to cycle_summary
The evaluator read only the score from InnerLoop's cycle_summary but
discarded scoring_method and test_details. The CLI then wrote its own
summary without test information, so cycle_summary.json always showed
heuristic scoring even when pytest pass rate was computed.
Now _read_cycle_summary returns the full dict and the evaluator includes
scoring_method + test_details in EvalResult.details, which the CLI
persists to the main project's cycle_summary.json.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: add outer loop guide and link from README
New docs/outer-loop.md covers the full outer loop architecture:
two-CEO model, pipeline (calibrate→evaluate→reflect→evolve→gate),
scoring (pytest pass rate + parsimony), worktree isolation, ephemeral
mode lifecycle, CLI reference, module map, and E2E findings.
Linked from the README/index.md under "Other Workflows".
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…mpt (akashgit#1294) The full assembled CEO prompt (33KB base + 13-31KB SKILL.md) exceeded Claude Code's ~40KB CLAUDE.md character limit in every workflow mode. Split the prompt delivery: CLAUDE.md now receives a slim ~2KB resume-resilient identity (Sacred Rules, agent dispatch syntax, review verdicts, mode pointer), while the full prompt continues via --append-system-prompt-file unchanged. Changes: - factory/models.py: Add prompt_core field to AgentRunRequest - factory/agents/runner.py: Add resolve_prompt_core() returning slim identity - factory/runners/claude.py: Write prompt_core to CLAUDE.md (fallback to full prompt for backward compat), backup/restore existing CLAUDE.md - factory/cli/_ceo_helpers.py: Pass prompt_core to interactive AgentRunRequest Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ite loop (akashgit#1298) * fix: update test helper to write declared artifacts, preventing infinite loop The _simulate_reloop_cycle() helper in TestLoopContextE2EComparison was writing generic {role}-latest.md files for all AgentNodes. After f33b35b reordered _detect_artifact to check node.writes before the generic reviews/ path, nodes with explicit writes (e.g. health_checker writing .factory/reviews/health-check.md) never had their declared artifacts created, so tool_next's artifact detection loop spun forever. Fix: when a node declares writes, write those exact paths instead of the generic fallback. Nodes without writes still get the old behavior. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update stale hardcoded count assertions to match current state Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: accept all valid convergence reasons in test_run_terminates_on_budget The engine's convergence behavior is nondeterministic — it can terminate with any reason from _get_convergence_reason(), not just the three previously asserted. This caused flaky test failures when the engine happened to hit diversity_collapse or another valid reason. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fix: write slim CEO identity to .claude/CLAUDE.md instead of full prompt
… docs/index.md (akashgit#1303) The old README.md (374 lines) had richer content than docs/index.md after PR akashgit#1100 turned README.md into a symlink. This restores the missing sections (Design Mode spec file tips, LangFuse tracing, Plugin install, Plugin Agents, Verified Skill Generation) while preserving all new content added since (mermaid diagrams, Other Workflows, Outer Loop, multi-issue focus examples). - Convert doc links from root-relative (docs/foo.md) to docs-relative (foo.md) - Use absolute GitHub URL for LICENSE badge and infra/langfuse cross-ref - Use absolute URL for logo image - Update 'uv run factory' commands to 'factory' Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Phase 1 of the repo cleanup plan — build a safety net before any deletion. Creates tests/test_smoke_cli.py (21 tests, <5 s) covering: - factory detect: all 5 ProjectState values - factory discover: eval profile generation against hello-cli fixture - factory study: observations.md written and non-empty - design_workflow: Tier 4 integration with mocked invoke_agent, real WorkflowExecutor, FnNode gates, and artifact file assertions - create_workflow: same pattern - factory agent <role>: 8 kept roles — prompt resolution + review file - factory refactory: workspace setup + session ID persistence Registers 'smoke' and 'e2e' pytest markers in pyproject.toml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes akashgit#1343
Changes
New file
tests/test_smoke_cli.py— 21 smoke tests covering all core factory modes:factory detect: all 5ProjectStatevalues tested with exact assertionsfactory discover: eval profile generation againsttests/fixtures/hello-cli, validatesEvalProfilemodelfactory study: observations file written and non-emptydesign_workflow(): Tier 4 integration — patchesinvoke_agentat the subprocess boundary, runs the realWorkflowExecutorwithFnNode/GateNodegates, assertsresearch-*.mdandcurrent.mdartifact files appearcreate_workflow(): same Tier 4 patternfactory agent <role>: 8 kept roles (researcher, strategist, builder, health_checker, code_reviewer, adversarial_tester, archivist, ceo) — verifies prompt resolution and.factory/reviews/<role>-latest.mdwrittenfactory refactory: workspace setup creates.refactory/,.claude/settings.local.json,CLAUDE.md; session IDs persist and reset correctlypyproject.toml: Registeredsmokeande2epytest markersPerformance:
pytest -m smokecompletes in ~4 seconds (target was <30s)This is Phase 1 of the repo cleanup plan — the safety net that must exist before any code deletion in subsequent phases.