fix: remove 17 dead workflow modes from definitions.py and registry (#1346) - #5
Draft
osilkin98 wants to merge 407 commits into
Draft
fix: remove 17 dead workflow modes from definitions.py and registry (#1346)#5osilkin98 wants to merge 407 commits into
osilkin98 wants to merge 407 commits into
Conversation
* feat: add --no-worktree flag to factory ceo and factory run Users testing in-flight branch changes had no way to run the CEO against their working tree — the factory always creates a worktree from the base branch. This flag skips worktree creation and cleanup, running the CEO in-place on whatever branch is checked out. When --no-worktree is set, wt_path = project_path and wt_branch = None. remove_worktree() calls are guarded to prevent shutil.rmtree() on the actual project directory. prune_stale() still runs regardless. * fix: add type-narrowing assertions for wt_branch in ceo.py mypy cannot correlate the no_worktree flag with the str | None type of wt_branch. Add assert statements before each remove_worktree() call to narrow the type within the existing guard blocks. * fix: restore deep-qa mode block dropped during rebase
* fix: update eval timeout values and coverage target - eval_tests: 120s → 600s (matches factory.md Test Timeout) - eval_coverage: 120s → 600s, --cov= → --cov=factory - eval_lint: 120s → 300s (safe margin) - eval_type_check: 120s → 300s (safe margin) - Update timeout error messages to match new values Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: scope eval tests and coverage commands to prevent timeout - eval_tests: Run specific test files with -x -q --tb=short for faster feedback - eval_coverage: Run subset of tests with coverage, parse actual percentage from output - Skip flaky BobAuth and preflight_error_unchanged tests Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: correct test file reference in eval/score.py Replace non-existent 'tests/test_eval.py' with 'tests/test_eval_growth.py' in the eval_tests command list. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: exclude telemetry platform test and remove -x flag from eval - Add test_interactive_sets_telemetry_platform_empty to -k exclusion filter (fails when TELEMETRY_PLATFORM env var is set) - Remove -x flag so all tests run even if one fails, giving more accurate score Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: add eval command to factory.md The Eval Command section had an empty bash code block, which prevented the factory eval system from knowing how to run the eval harness. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * fix: persist trace_id.txt across worktree cleanup Before cleanup removes a worktree directory, copy telemetry files (trace_id.txt) from the worktree's .factory/ to the main project's .factory/. When .factory/ is a symlink (the default), files are already in the right place and no copy is needed. Fixes benchmark script's find command failing to locate trace_id.txt after CEO exit. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
* refactor: rename GRAPH-SPEC.md to SPEC.md GRAPH-SPEC.md was a historical name from when it coexisted with a separate human-authored SPEC.md design doc. Now that the machine- generated behavioral spec is the canonical spec, simplify the name. The old abstract SPEC.md (meta-harness design spec) is superseded by the concrete behavioral spec and removed. * fix: update stale GRAPH-SPEC.md reference in SPEC.md
The CEO and specialist agents run as Claude Code subprocesses, which gives them access to Claude Code's native Agent tool. This caused the CEO to sometimes spawn subagents via the native tool instead of `factory agent <role>`, bypassing prompt resolution, playbook injection, review file capture, event emission, and telemetry. Add `--disallowedTools Agent` to all Claude Code invocation paths (headless, interactive, background, tmux) and reinforce the prohibition in the CEO prompt's Forbidden Actions list. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Address review feedback on PR #997: the refactory supervisor path (cmd_refactory) was missing the --disallowedTools Agent flag, and the background and tmux command construction paths lacked dedicated tests. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
fix: disallow Claude Code native Agent tool in factory subprocesses
The background and tmux disallowed-tools tests used asyncio.get_event_loop().run_until_complete() inside sync functions, which fails silently under pytest-asyncio's running event loop. The mocks were never called, causing IndexError on empty call_args_list. Convert both to async test functions (pytest-asyncio auto mode handles them) and await the async functions directly. Also mock asyncio.sleep in the background test to avoid a 5-second wait from the polling loop. Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…tor, remove --max-turns cap (#1000) - Increase AgentNode.timeout from 1200s to 7200s in all 5 benchmark workflows (featurebench, swebench, terminalbench, legacybench, programbench) so agents have enough time for complex tasks. - Forward node.timeout from workflow definitions to invoke_agent in the executor's _run_agent method. Previously the node timeout was silently dropped and the hardcoded default of 600s always won. - Bump max_timeout wall-clock backstop from 3600s to 14400s so it doesn't kill agents before their configured timeout expires. - Remove --max-turns 1000 from Claude Code CLI invocation, letting Claude Code use its own default. Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…nvocations (#1001) * fix: convert disallowed-tools tests to async to fix IndexError The background and tmux disallowed-tools tests used asyncio.get_event_loop().run_until_complete() inside sync functions, which fails silently under pytest-asyncio's running event loop. The mocks were never called, causing IndexError on empty call_args_list. Convert both to async test functions (pytest-asyncio auto mode handles them) and await the async functions directly. Also mock asyncio.sleep in the background test to avoid a 5-second wait from the polling loop. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: design mode routing preserves ceo_mode='design' for existing projects The ternary at ceo.py:580 incorrectly mapped design_existing=True to ceo_mode='build', causing the CEO to receive Build mode instructions instead of reading skills/workflow-design/SKILL.md. Replace with an if/elif chain that preserves design mode routing. Also adds a diagnostic warning when .factory/ exists without config.json to guide users toward running 'factory init'. Closes #999, addresses #908 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: design mode routing uses mode=='design' for all design invocations The previous fix only preserved ceo_mode='design' for design_existing (existing projects). New ideas via --mode design still fell through to ceo_mode='build'. Now `elif mode == "design"` catches both cases, ensuring the design workflow (with its user approval gate) is always used when the user passes --mode design. Also updates the design_idea task string to remove the incorrect "proceed to Build mode" language, and adds a test verifying that research_ideation correctly routes to 'build'. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…1003) Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…arks (#1005) Propagate FACTORY_BENCHMARK and FACTORY_INSTANCE_ID env vars through the benchmark pipeline into Langfuse trace metadata, enabling deterministic trace matching. Fix find_matching_trace to prefer metadata-based filtering, remove the dangerous all-traces fallback, and use earliest-timestamp selection instead of max-latency tiebreaker. Closes #1004 Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…1010) Add an optional `notes` field to FnNode (default empty string) so that generated SKILL.md files include prose context before each bash command block. This closes the gap where FnNode was the only node type missing contextual prose — AgentNode has prompt_template, Study has hardcoded prose, GateNode has gate_prompt. Populate all 28 FnNode instances in definitions.py with concise notes explaining what the command does, required CEO substitution variables, and ordering constraints. Closes #1009 Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* fix: preserve Harbor exception details in benchmark CI artifacts When the factory agent crashes before creating a Langfuse trace, Harbor writes crash details to exception.txt in the trial directory — but cleanup() destroys these before CI artifact upload, making crashes undiagnosable. Extract exception.txt and trial.log from JOBS_DIR before rm -rf, include the exception text in DETAILS_JSON, and surface it in analyze_failure.py when no Langfuse trace exists. Closes #1011 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: pass DETAILS_JSON via env var to prevent multi-line exception escaping bug write_result() in lib.sh used shell interpolation inside a Python string literal (_dj = '${DETAILS_JSON}'), which caused Python to reinterpret \n escape sequences from json.dumps as actual newlines, breaking json.loads for multi-line exceptions (stack traces). Now DETAILS_JSON is exported as an env var and read via os.environ.get(), bypassing the string parser. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…#1014) (#1015) Enhance gate_verify from a simple todos-only check to a comprehensive todos → compilation → multi-tier test pipeline. The gate now runs compile.sh with timeout 7200, then probes for test infrastructure (make test, pytest, test.sh) and writes structured results to /workspace/test-results.txt. Builder and reviewer prompts updated to reference test results on RELOOP iterations. Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
* docs: add benchmark contribution guide Create docs/contributing-benchmarks.md with a comprehensive walkthrough for contributing benchmarks, covering all 18 linter validation conditions, Harbor execution, CI integration, and a pre-submission checklist. Update README.md and docs/contributing.md with cross-references. Add the new page to mkdocs.yml nav config. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove prose dashes from benchmark contribution guide Change heading 'Pre-Submission Checklist' to 'Submission Checklist' and replace 're-exports' with 'exports' to satisfy the no-dashes constraint. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Add harborindex as a new benchmark following existing patterns. Uses the generic factory ceo approach (no workflow override) since Harbor-Index is a diverse meta-benchmark. Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…action (#1023) The CEO's workflow playbook (SKILL.md) was previously loaded as a conversation message — vulnerable to lossy summarization during context compaction. This moves it into the system prompt where it is re-injected every turn and survives compaction. Changes: - Add workflow_mode parameter to resolve_prompt(), invoke_agent(), and run_ceo_with_completion_guard() - When role is "ceo" and workflow_mode is set, append the corresponding SKILL.md content to the system prompt with a clear heading - Update all call sites in cmd_ceo() (interactive, headless, review, qa, deep-qa modes) and _run_single_cycle() to pass workflow_mode - Remove "read skills/workflow-{mode}/SKILL.md" instructions from _build_ceo_task() — the playbook is already in the system prompt - Add tests for SKILL.md injection and task string cleanup Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
…e exists (#1021) * fix: use Harbor artifacts in analyze_failure.py when no Langfuse trace exists Remove the claude-code solver short-circuit that produced dead-end output. Instead, all solvers flow through generate_report() which now discovers trial.log files adjacent to result JSON and combines them with details.exception for LLM analysis or raw rendering under "Harbor Artifacts". Summary mode also falls back to trial.log content when no trace is available. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: capture Harbor output to log file for failure analysis when no Langfuse trace exists Harbor's stdout/stderr was lost to the terminal. Now tee captures it to a log file. When trial.log isn't found in the jobs directory, the Harbor log is copied to the trial.log artifact path so analyze_failure.py picks it up automatically. Uses PIPESTATUS to preserve Harbor's real exit code through tee. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
update README installation instructions to be global
* feat: add ToM-SWE benchmark workflow and Harbor adapter (#1019) Add ToM-SWE (ICML 2026) benchmark support — preference-aware task solving under deliberately vague instructions with embedded user profiles. - factory/workflow/contributed/tomswe/: 4-node pipeline (study → builder → gate_verify → auto_merge) with builder prompt adapted for vague instructions and user preference alignment - benchmarks/factory_harbor_agent.py: TomsweFactoryCeo class - benchmarks/config.sh: tomswe benchmark config entry - factory/workflow/definitions.py: register tomswe workflow (22 total) - SPEC.md: update workflow counts from 20 to 22 - 24 tests across 5 test classes, all passing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: exclude terminal benchmark workflows from QA enforcement tests Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * feat: add ToM-SWE local Harbor task and update runner config Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: remove pre-init git from tomswe Dockerfile FactoryCeo.run() handles git initialization. Pre-initializing git in the Dockerfile conflicts with the agent's setup phase. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: configure git safe.directory in tomswe Dockerfile for Harbor Harbor's exec_as_root install phase creates files as root in /workspace, then exec_as_agent runs git as the agent user. Git 2.35.2+ refuses to operate on repos with ownership mismatches unless safe.directory is set. Verified E2E: `FACTORY_GIT_REF=$(git rev-parse HEAD) benchmarks/run.sh tomswe discount-calc` → RESOLVED (1/1), reward 1.0, runtime 2m51s. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add 4 more ToM-SWE sample tasks for E2E validation Add sort-order, date-parse, dedup-list, and csv-export tasks with varied user profiles (verbose/concise, pytest/unittest, functional style, data engineer). All 5 tasks verified: 5/5 resolved, 100% accuracy, 4m09s total runtime with concurrency=5. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: make ToM-SWE benchmark CI-interoperable via swe-bench dataset reuse Override run() in TomsweFactoryCeo to inject deterministic user profiles (hash-based selection from 15 profiles) into task instructions before solving. Switch tomswe config from local sample tasks to swe-bench/swe-bench-verified dataset. Add tomswe matrix entries (factory + claude-code solvers) to benchmark CI workflow. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Write full CEO prompt to .claude/CLAUDE.md in the worktree so it survives session transitions (background via ←, resume, daemon restart). Move --disallowedTools Agent to .claude/settings.local.json since CLI flags are not carried over on transitions. Add a session guard to remove_worktree() that checks for active background sessions before deleting the worktree directory. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
fix: Make CEO prompt resilient to session transitions
…992) * feat: add parallel-improve workflow for concurrent experiment execution Introduces a new `parallel-improve` workflow that runs N hypotheses concurrently in isolated git worktrees, then selects the best result via tournament-style selection. This is a stepping stone toward integrating parallel experimentation into the core improve loop. New primitives: SubgraphForkNode (fan-out subgraphs into worktrees), SelectionNode (compare and pick the best experiment). The executor spawns independent WorkflowExecutor instances per branch for full isolation. Adds ParallelConfig model, "superseded" verdict type, experiment worktree support, and 31 new tests. Ref: #987 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update workflow registry count for parallel-improve Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * test: increase patch coverage for parallel-improve workflow Add 39 tests covering the previously untested code paths flagged by Codecov: _execute_selection non-dry-run (score comparison, merge failure, cleanup error tolerance), _execute_subgraph_fork error handling and non-dry-run paths, _parse_parallel config parsing, ExperimentStore superseded verdict roundtrip, create_experiment_worktree lifecycle, prune_stale exp- prefix handling, and SubgraphForkNode validation errors. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * Merge upstream/main into issue-987-parallel-improve-workflow Resolves conflict in tests/test_skill_export.py: upstream removed QA_EXEMPT_WORKFLOWS, replaced with dynamic _workflows_with_builder() filter that excludes SubgraphForkNode workflows (QA runs inside the subgraph, not in the skill prose). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: isolate .factory/ in experiment worktrees to prevent shared eval state create_experiment_worktree() was symlinking .factory/ to the project's shared directory, causing all parallel experiment branches to read/write the same last_eval.json. The selection node then compared identical scores, making branch selection effectively random. Now each experiment worktree gets its own .factory/ directory seeded with config files (config.json, eval_profile.json, strategy/, agents/) but NOT mutable state (results.tsv, experiments/, last_eval.json), so parallel eval results stay independent. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: update register_all workflow count to 23 after ToM-SWE merge Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
The generated plugin package is a standalone artifact that stays in the output directory. Remove the "commit and open PR" instruction from the plugin path in both the builder prompt and the task builder output. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
docs: add plugin mode examples to CLAUDE.md
* feat: add multi-benchmark support to outer loop (#1325) Add pluggable test output parsers (pytest, exit_code, json, exact_match), TOML-based benchmark config registry, instance preparation tooling, and full pipeline wiring. All existing FeatureBench behavior preserved via backward-compatible defaults and import aliases. - Phase 1: factory/outer_loop/evaluators/ package with 4 format parsers - Phase 2: BenchmarkConfig + TOML registry (featurebench, swebench, aime) - Phase 3: Wire config into CLI, SwarmEngine, SwarmEvaluator, InnerLoop - Phase 4: Instance preparation with prep_command and validation - Phase 5: 60 E2E tests covering 3 benchmarks (featurebench, swebench, custom) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: correct exact_match and JSON metric_path in InnerLoop._parse_test_output Fix 1: exact_match format now reads expected answer from expected_answer.txt or expected.txt in the project directory instead of using stderr (which contains error messages, not expected answers). Fix 2: JSON format now uses a configurable metric_path (dotted path lookup) instead of hardcoded 'score'/'pass_rate' keys. Added metric_path parameter to InnerLoop, FeatureBenchInnerLoop, SwarmConfig, and wired through SwarmEvaluator. Updated test_custom_inner_loop_json_parsing to assert score==0.92 with metric_path='accuracy'. Added 4 tests for exact_match file-based lookup. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: wire seed_workflow to SwarmEngine.seed() and fix prep_command shell operators Fix 1: SwarmEngine.seed() now checks config.seed_workflow and looks up the workflow from WorkflowRegistry before falling back to the passed-in base_workflow. This completes the wiring from benchmark TOML configs through to the evolutionary search seeding. Fix 2: prepare_instances() now detects shell operators (&&, ||, ;, |) in prep_command and uses shell=True when present, so chained commands work. Updated swebench.toml to use only supported template vars ({instance_id}, {instance_dir}) instead of unsupported {repo_url} and {commit}. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * feat: add ForecastBench benchmark config Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add ForecastBench to built-in benchmarks and add Working with Your Benchmark section Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
No workflow or code path references this 38-line prompt. Closes #1327 Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: remove unused evolver.md agent prompt No workflow or code path references this 38-line prompt. Closes #1327 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * refactor: remove dead runners (bob, codex, opencode) Remove three unused runner backends and their supporting code (~3,875 lines). The factory now uses Claude Code as its sole runner. Deleted: bob.py, codex.py, opencode.py, usage.py, test_codex_runner.py, test_opencode_runner.py. Cleaned references from 13 source and test files. Updated CLAUDE.md runner documentation. Closes #1326 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: refresh mkdocs theme — monochrome palette, clean dark mode, light/dark logo - Swap palette from stock deep purple/amber to black primary with Inter/JetBrains Mono fonts - Dark mode: true navy background (#0B1220), matching header, sidebar, and code blocks - Light mode: AA-compliant link blue (#0B6BDE), tinted code blocks (#F4F7FB) - Create transparent logo variants (logo-light.png, logo-dark.png) with Material #only-light/#only-dark swap - Strip admonitions to minimal left-border style, remove shadows and icons - Two-color header wordmark via CSS pseudo-elements (re in blue, :factory in light gray) - Hide default Material book icon, cap logo width at 480px - Remove navigation.expand to collapse sidebar sections - Trim TOC to depth 2 - Normalize badges: CI (default signaling), Python, MIT License, Runner: Claude Code - Remove redundant "Full Documentation" link from docs homepage Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: resolve two mypy type errors in outer_loop 1. json_evaluator.py: suppress arg-type error on float(value) where value is typed as object but is guarded by try/except at runtime. 2. featurebench_inner_loop.py: widen _evaluator type from FeatureBenchEvaluator to Evaluator protocol so the assignment from get_evaluator() is compatible. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: extend codecov ignore list to cover docs and outer_loop evaluators Adds docs/, factory/outer_loop/evaluators/, and factory/outer_loop/featurebench_inner_loop.py to the codecov ignore list to fix patch coverage check failures on files that don't need coverage tracking. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add documentation links to docs/index.md Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: add docs badge back to badge row Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: trim badges to Python, License, Docs Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ows (#1300) Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add smoke test harness for core factory modes (#1343) Phase 1 of the repo cleanup plan — build a safety net before any deletion. Creates tests/test_smoke_cli.py (21 tests, <5 s) covering: - factory detect: all 5 ProjectState values - factory discover: eval profile generation against hello-cli fixture - factory study: observations.md written and non-empty - design_workflow: Tier 4 integration with mocked invoke_agent, real WorkflowExecutor, FnNode gates, and artifact file assertions - create_workflow: same pattern - factory agent <role>: 8 kept roles — prompt resolution + review file - factory refactory: workspace setup + session ID persistence Registers 'smoke' and 'e2e' pytest markers in pyproject.toml. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add 6 deletion safety test classes to smoke test suite Adds TestPackageImports, TestCommandHelp, TestRegistryInstantiation, TestPromptResolution, TestSuiteCollects, and TestNoDanglingReferences to catch broken imports, dead CLI commands, missing prompt files, and stale symbol references before they reach production. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: pass cwd to nested pytest in TestSuiteCollects The test_collect_only test runs `pytest --collect-only` as a subprocess. In CI with parallel workers (pytest-xdist), the subprocess inherits a worker-specific cwd where no tests exist, causing rc=5 (no tests collected). Fix by explicitly setting cwd to the repo root. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* docs: decouple README.md from docs/index.md symlink, share content via snippets Break the README.md → docs/index.md symlink that has caused recurring issues (PRs #1100, #1303). README.md is now a standalone file with a GitHub-compatible header (HTML badge tags, raw PNG logo URL) while docs/index.md keeps the MkDocs-specific header (light/dark logo swap, markdown attribute badges). Shared body content lives in README.md between pymdownx.snippets section markers, and docs/index.md pulls it in via --8<-- "README.md:body". Single source of truth, two renderer-appropriate headers. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * docs: use relative docs/ links in README body for dual-context compat Convert absolute hosted-docs URLs back to relative docs/ paths (e.g. docs/architecture.md). These work on GitHub from the repo root. The existing hooks/link_rewrite.py strips the docs/ prefix at MkDocs build time so they resolve correctly in the docs site too. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: rewrite hook to on_page_content for snippet-included links The hook now runs after pymdownx.snippets expands the README body, rewrites href="docs/foo.md" → href="foo/" in rendered HTML. Also adds min-width to header wordmark for clickable home link. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: downgrade link validation to info so --strict passes The docs/ prefixed links in the snippet-included README body are rewritten by the on_page_content hook, but MkDocs validates them at the markdown level before the hook fires. Downgrade not_found link validation from warning (fatal in strict) to info. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: pre-render link rewrite so mkdocs --strict passes without downgrading validation Replace on_page_content (post-render) hook with on_page_markdown (pre-render) that manually expands the README.md snippet and rewrites docs/ links before MkDocs link validation runs. Remove the not_found: info workaround. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
npm install -g may skip the native binary optional dependency. Running install.cjs manually ensures the platform binary is downloaded. Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
factory/workflow/executor.py - _parse_agent_verdict: recognize explicit PROCEED; unparseable/empty last line now halts with parse-failure reason instead of proceeding - _parse_fn_verdict: recognize pass/proceed; unparseable/empty/malformed-JSON output now halts with parse-failure reason instead of proceeding - _evaluate_gate: fn gate with no evaluator_command halts with misconfiguration reason instead of proceeding tests/test_workflow_executor.py - Add TestGateVerdictFailClosed: 17 regression cases for fail-closed parsing + no-command fn gate
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* fix: bootstrap factory.md and config.json in design mode HALT path Design mode's HALT path (no existing config.json) now creates factory.md and config.json via three new nodes: gate_factory_md_exists, create_factory_md, and factory_init. A gate checks if factory.md already exists to avoid overwriting it. Also syncs bootstrap artifacts (config.json, eval_profile.json, factory.md) from worktree back to main project on cleanup, so they survive worktree removal. Closes #1270 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: update plan workflow tests for bootstrap nodes The plan workflow (design_workflow(just_plan=True)) inherits the bootstrap nodes added in the design mode fix. Update expected node count (18->21), edge count (23->27), and edge list. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * test: add coverage for _sync_bootstrap_to_main Four tests covering: fresh file copy, symlink skip, existing file skip, and no-op when .factory/ is absent. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
ci: add crqu to CEO review allowed users
…1366) The test_plateau_detection assertion only accepted 3 of 6 valid convergence_reason values, causing flaky failures when the engine terminated via diversity_collapse, target_score_reached, or unknown. Also fixed the same narrow pattern in test_coverage_gaps.py. Closes #1363 Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ky CI downloads (#1362) * fix: mock ONNX embedding function in mempalace tests to eliminate flaky CI downloads Extends the `isolated_palace` fixture to monkeypatch `mempalace.embedding.get_embedding_function` with a deterministic, hash-based fake that requires no network access or ONNX runtime. Uses SHA-256 seeded PCG64 RNG to produce distinct float32 vectors per input text (dimension 384, matching MiniLM output shape), preserving semantic search test validity. Closes #1314 Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: parse agent gate verdict from first line when last line has no verdict keyword Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* refactor: extract SkillOpt to RobotSail/skillopt, clean up this repo (#1342) Phase 1 (separate repo): Created RobotSail/skillopt with full commit history via git filter-repo. Set up pyproject.toml (hatchling + hatch-vcs), CI (uv-based, pytest + ruff, Python 3.11-3.13 matrix), and renamed factory.skillopt imports to skillopt. Phase 2 (this repo cleanup): - Delete factory/skillopt/ files: reflect, trainer, gate, clip, aggregate, failure_tracker, slow_update, skill, types, adapter, adapters/, prompts/, __main__ - Keep factory/skillopt/yaml_surface.py and __init__.py (interop seam) - Split tests/test_skillopt.py: extract yaml_surface tests to tests/test_yaml_surface.py, delete original and adapter/integration tests - Remove skill-refine workflow from definitions.py registry - Remove skill-refine from WORKFLOW_META in skill_export.py - Remove SKILL_REVIEWER from AgentRole enum and AGENT_CONFIGS - Delete factory/agents/prompts/skill_reviewer.md - Update test assertions in test_lazy_loading.py and test_workflow_definitions.py Closes #1342 * fix: update expected workflow count to 35 after skill_refine removal Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* feat: add smoke/integration test harness for core CLI surfaces (#1343) - Register smoke and e2e markers in pyproject.toml with --strict-markers - Apply @pytest.mark.e2e to test_workflow_e2e.py and test_tmux_e2e.py - Mark test_install.py with @pytest.mark.slow - Create tests/test_smoke_cli.py (18 tests): - factory detect: 5 parametrized cases for each ProjectState - factory discover: validates EvalProfile written and parseable - factory study: asserts observations.md non-empty - factory workflow run design/create: Tier 4 dry-run via WorkflowExecutor - factory agent: 8 roles via cmd_agent with mocked invoke_agent - factory refactory: workspace setup validation - Add CI smoke job in ci.yml as required gate before full test suite Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: use strict_markers ini option and add 60s timeout to test_install The `addopts = ["--strict-markers"]` setting was silently ignored by pytest 9.0.3 — unregistered markers produced warnings instead of collection errors. Replace with `strict_markers = true` as a direct ini option which works correctly. Add pytest-timeout dev dependency and a 60-second timeout to test_install.py's pytestmark to prevent install tests from hanging. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Remove 11 files with no remaining imports or references:
- factory/templates/{score.py,__init__.py,factory_config.md}
- factory/compress/{__init__.py,evaluator.py,inner_loop.py,outer_loop.py}
- factory/agents/prompts/{reflector.md,evolver.md}
- tests/{test_template_score.py,test_compress_inner_outer.py}
Surgically remove TestFactoryConfigTemplate class and TEMPLATES_DIR
fixture from tests/test_prompts.py (depended on deleted template).
Preserves factory/templates/design_checks/ (still referenced by
frontend_design_scan_workflow, deferred to #1346).
Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…mnants (#1375) * fix: complete runner cleanup — remove remaining codex/bob/opencode remnants (#1345) PR #1334 removed the core runner implementations. This commit completes the cleanup by removing the remaining ~400 lines of codex plugin generation code, 280 lines of dead docs, and scattered string references across 19 files. Deleted: - docs/codex-mcp.md (78 lines, codex-specific MCP docs) - docs/runner-v2-spec.md (202 lines, dead multi-runner spec) Cleaned: - factory/agents/plugin.py: removed generate_codex_agent_toml(), check_codex_agents_in_sync(), _sandbox_mode(), _escape_toml_multiline_literal(), and associated role sets - scripts/sync_agents.py: removed codex imports and TOML generation logic - factory/agents/runner.py: updated error message (was BOBSHELL_API_KEY-specific) - factory/ceo_completion.py: updated BobRunner comment - factory/podman.py: updated dry-run convention comment - docs/setup.md: removed Bob Shell env var section - docs/configuration.md: removed bob profile example, bob env vars - docs/contributing.md: removed Codex from multi-backend list - .github/workflows/plugins.yml: removed codex-agents/ from git add - .gitignore: removed /codex-agents/ entry - tests: removed codex/bob/opencode-specific test data and imports Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> * fix: remove 3 stale 'bob' docstring references in runners Replace remaining 'bob' examples with 'claude' in docstrings: - protocol.py: "(claude, bob, etc.)" → "(e.g. claude)" - _stream.py: b"[bob:researcher] " → b"[claude:researcher] " - _stream.py: "[bob:researcher]" → "[claude:researcher]" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Oleg Silkin <97077423+RobotSail@users.noreply.github.com> Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…1346) Delete 17 workflow functions (~3,300 lines) from definitions.py, their registry entries, WORKFLOW_META entries in skill_export.py, MODE_PHASES/ MODE_AGENT_TO_PHASE/MODE_EVENT_TO_PHASE entries in visualizer/state.py, and all associated artifacts. Deleted workflow functions: improve, research, meta, discover, review, refine, skill_refine, doc_generate, doc_update, spec_update, frontend_design, frontend_design_scan, frontend_design_discover, evolve, parallel_improve, founder, study_standalone. Surviving modes: design, create, spec-generate (plus all contributed benchmark workflows untouched). Kept as library functions (not registered): build_workflow (called by design_workflow), _study_subgraph, _deep_qa_subgraph, _research_subgraph (imported by deep_qa.py and research.py). Synced all 6 locations: registry, WORKFLOW_META, MODE_*_TO_PHASE, _auto_detect_mode, --mode help text, CEO routing tables. Deleted artifacts: factory/agents/prompts/frontend_design/ (13 files), factory/templates/design_checks/ (6 scripts), 6 test files for deleted modes, 18 skill directories (generated, not tracked). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_workflow_integration.py: use 'archivist_build' node (build_workflow has archivist_build/archivist_plan, not 'archivist') - test_workflow_qa.py: remove test_registered checking deep-qa in registry (deep-qa was intentionally removed from register_all) - test_cycle_analyzer.py: use 'researcher_similar' node (build_workflow uses forked researcher nodes, not a single 'researcher') Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…moval Update test assertions across 5 files to match the reduced workflow registry after removing 17 dead modes in the prior commit: - test_prompts.py: Replace build/improve workflow refs with design workflow (16 tests fixed) - test_visualizer.py: Replace improve/research/build/meta mode-specific phase tests with design mode tests (16 tests fixed) - test_lazy_loading.py: Update expected workflow set, replace deep-qa with swebench for lazy import test (3 tests fixed) - test_workflow_research.py: Replace registration assertions with not-registered check (2 tests fixed) - test_dashboard.py: Replace improve mode phase-detail test with design mode (1 test fixed) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
osilkin98
force-pushed
the
factory/run-3427af8a
branch
from
August 21, 2026 19:07
c6f9f6e to
5ed3499
Compare
Clean up 16 lint errors across 11 files introduced during rebase: - Remove unused imports (build_workflow, FnNode, register_all) - Remove stale noqa: F401 directives on symbols that are now used - Remove noqa directives for non-enabled rules (ANN001, ANN202, BLE001, PLW0603, F841) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- test_cli.py: remove templates/ dir assertion since factory/templates/ is now empty after design_checks/ deletion - test_plan_workflow.py: remove test_plan_workflow_validates which calls `factory workflow validate plan` — plan is no longer a separate registry entry (produced via design_workflow(just_plan=True) instead) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes akashgit#1346
Changes
_get_builtin_registry(). Surviving entries: design, create, spec-generate, plus all contributed benchmark workflows._auto_detect_mode()to route all ProjectState values to "design". Removed unused_has_research_targetimport. Removedworkflow_modeargs from review/deep-qa handlers (these modes construct tasks inline, not via SKILL.md).--modehelp text to list only surviving modes (design, create)..claude/CLAUDE.mdandfactory/agents/prompts/ceo.mdto route all states to design mode.factory/agents/prompts/frontend_design/(13 prompt files),factory/templates/design_checks/(6 shell scripts), 18 generated skill directories (not tracked in git).Safety constraints verified
build_workflow()function KEPT (called bydesign_workflow()) — only registry entry removed_study_subgraph(),_deep_qa_subgraph(),_research_subgraph()KEPT — imported bydeep_qa.pyandresearch.pyfactory/refactory.pyUNTOUCHED