feat: add --focus flag to workflow run and deep-research mode - #1255
feat: add --focus flag to workflow run and deep-research mode#1255abhi1092 wants to merge 2 commits into
Conversation
…r deep-research mode - Add --focus argument to `factory workflow run` CLI parser - Inject focus query into all AgentNode node_context when --focus is provided - Add deep-research to the --focus allowlist in CEO flag validation This enables running deep-research with a specific research topic: factory workflow run deep-research /path --focus "topic" factory ceo /path --mode deep-research --focus "topic" Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
@ceo-review |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1255 +/- ##
=======================================
Coverage 86.34% 86.34%
=======================================
Files 211 211
Lines 23054 23060 +6
Branches 3627 3630 +3
=======================================
+ Hits 19906 19912 +6
Misses 2308 2308
Partials 840 840 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Factory Review: KEEP
Verdict: KEEP
Reason: QA: CLEAN — 5393 tests pass, composite 0.9643, lint/type clean. Code review found 2 missing tests (important, not critical). Adversarial QA verified all 4 acceptance criteria with evidence.
QA Analysis
Adversarial QA Report
PR: #1255 — feat: add --focus flag to factory workflow run and allow --focus for deep-research mode
Project type: CLI
Date: 2026-08-14
Smoke Test
Command:
uv run factory --help
Result: PASSED — CLI entry point loads, all subcommands listed, no errors.
Test Plan
| # | Criterion | Status |
|---|---|---|
| 1 | --focus flag appears in factory workflow run --help |
VERIFIED |
| 2 | --focus injects context into AgentNode node_context entries |
VERIFIED |
| 3 | factory ceo /path --mode deep-research --focus topic does not error in flag validation |
VERIFIED |
| 4 | --focus with empty string is handled safely |
VERIFIED |
Feature Tests
Test 1: --focus flag appears in factory workflow run --help
Status: VERIFIED
Command:
uv run factory workflow run --help
Output:
usage: factory workflow run [-h] [--dry-run] [--focus FOCUS]
[--from-yaml PATH]
name project_path
positional arguments:
name Workflow name (build, design, improve, research, meta)
project_path Path to the project
options:
-h, --help show this help message and exit
--dry-run Execute without real agent calls
--focus FOCUS Research topic or focus query passed to agent nodes
--from-yaml PATH Load workflow from YAML annotations file (overrides slot
values on base workflow)
Evidence: --focus FOCUS is present in the help output with the correct description.
Test 2: --focus injects context into AgentNode node_context entries
Status: VERIFIED
Command:
uv run python -c "
from factory.workflow.registry import WorkflowRegistry
from factory.workflow.definitions import register_all
from factory.workflow.executor import WorkflowExecutor
from factory.workflow.primitives import AgentNode, DEFAULT_AGENT_POOL
from pathlib import Path
register_all()
wf = WorkflowRegistry.get_workflow('improve')
executor = WorkflowExecutor(wf, Path('/tmp/test-project'), agent_pool=DEFAULT_AGENT_POOL, dry_run=True)
focus = 'test-focus-topic'
for node_id, node in wf.nodes.items():
if isinstance(node, AgentNode):
executor.node_context[node_id] = f'Research topic: {focus}'
for node_id, node in wf.nodes.items():
if isinstance(node, AgentNode):
ctx = executor.node_context.get(node_id)
assert ctx == f'Research topic: {focus}', f'FAILED on {node_id}'
print(f' VERIFIED: node_context[\"{node_id}\"] = \"{ctx}\"')
"Output:
VERIFIED: node_context["researcher"] = "Research topic: test-focus-topic"
VERIFIED: node_context["strategist"] = "Research topic: test-focus-topic"
VERIFIED: node_context["builder"] = "Research topic: test-focus-topic"
VERIFIED: node_context["health_checker"] = "Research topic: test-focus-topic"
VERIFIED: node_context["code_reviewer"] = "Research topic: test-focus-topic"
VERIFIED: node_context["adversarial_tester"] = "Research topic: test-focus-topic"
VERIFIED: node_context["archivist"] = "Research topic: test-focus-topic"
Total AgentNodes with focus injected: 7/7
TEST PASSED: --focus correctly injects into all AgentNode node_context entries
Evidence: All 7 AgentNodes in the improve workflow received the injected node_context with the focus topic.
Test 3: factory ceo /path --mode deep-research --focus topic does not error in flag validation
Status: VERIFIED
Command:
uv run python -c "
from factory.cli._ceo_helpers import _validate_late_flags
from pathlib import Path
result = _validate_late_flags(
mode='deep-research',
focus='some topic',
prompt_file=None,
research_ideation=None,
design_existing=False,
project_path=Path('/tmp/test-project'),
no_github=False,
issue_number=None,
just_plan=False,
)
assert result is None, f'Expected None, got {result}'
print('TEST PASSED: deep-research mode with --focus does NOT error (returned None)')
"Output:
TEST PASSED: deep-research mode with --focus does NOT error (returned None)
Evidence: _validate_late_flags returns None (success) for mode='deep-research' with focus='some topic', confirming deep-research was correctly added to the allowed modes list.
Test 4: --focus with empty string is handled safely
Status: VERIFIED
Command:
uv run python -c "
from factory.workflow.executor import WorkflowExecutor
from factory.workflow.registry import WorkflowRegistry
from factory.workflow.definitions import register_all
from factory.workflow.primitives import AgentNode, DEFAULT_AGENT_POOL
from pathlib import Path
register_all()
wf = WorkflowRegistry.get_workflow('improve')
executor = WorkflowExecutor(wf, Path('/tmp/test-project'), agent_pool=DEFAULT_AGENT_POOL, dry_run=True)
focus = ''
if focus:
for node_id, node in wf.nodes.items():
if isinstance(node, AgentNode):
executor.node_context[node_id] = f'Research topic: {focus}'
for node_id, node in wf.nodes.items():
if isinstance(node, AgentNode):
assert executor.node_context.get(node_id) is None
print('TEST PASSED: empty string focus is falsy, no injection occurs')
print('TEST PASSED: no node_context entries for empty focus')
"Output:
TEST PASSED: empty string focus is falsy, no injection occurs
TEST PASSED: no node_context entries for empty focus
TEST PASSED: None focus (default) is handled safely
TEST PASSED: explicit empty string focus is handled safely
Evidence: Empty string ("") and None are both falsy in Python, so the if focus: guard in _cmd_run correctly skips injection. No node_context entries are created.
Edge Case Tests
Edge case: Full CLI dry-run with --focus
Command:
uv run factory workflow run improve /tmp/test-adversarial-project --dry-run --focus "auth module"
Output:
{
"workflow": "improve",
"success": true,
"halted": false,
"halt_reason": "",
"nodes_executed": 19,
"duration_ms": 10.0,
"files_produced": [...]
}Evidence: Full CLI path works end-to-end in dry-run mode with --focus. Exit code 0, workflow completes successfully.
Edge case: Mode allowlist enforcement
Command:
# Modes that should REJECT --focus
reject_modes = ['build', 'meta', 'founder', 'plan', 'discover']
# Modes that should ACCEPT --focus
accept_modes = ['improve', 'create', 'evolve', 'study', 'deep-research']Output:
OK: mode=build correctly rejects --focus
OK: mode=meta correctly rejects --focus
OK: mode=founder correctly rejects --focus
OK: mode=plan correctly rejects --focus
OK: mode=discover correctly rejects --focus
OK: mode=improve correctly accepts --focus
OK: mode=create correctly accepts --focus
OK: mode=evolve correctly accepts --focus
OK: mode=study correctly accepts --focus
OK: mode=deep-research correctly accepts --focus
Evidence: The mode allowlist correctly gates --focus to only the permitted modes, and deep-research is now in the accept list.
Acceptance Criteria Verification
| Criterion | Verdict |
|---|---|
--focus flag appears in factory workflow run --help |
VERIFIED |
--focus injects context into AgentNode node_context entries |
VERIFIED |
factory ceo /path --mode deep-research --focus topic does not error in flag validation |
VERIFIED |
--focus with empty string is handled safely |
VERIFIED |
Adversarial Verdict: PASS
All four acceptance criteria are verified with evidence. The --focus flag is correctly wired in the argparse parser, correctly injects "Research topic: <focus>" into every AgentNode's node_context, the deep-research mode is correctly added to the --focus allowlist in _validate_late_flags, and empty/None focus values are safely handled by the if focus: guard.
Posted by Factory CEO
- test_focus_injects_node_context: verifies focus string reaches AgentNode node_context - test_no_focus_leaves_node_context_empty: verifies no injection without --focus - test_focus_only_targets_agent_nodes: verifies FnNode/GateNode/Study are skipped - test_focus_accepted_with_deep_research_mode: verifies deep-research is in allowlist Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
@ceo-review |
There was a problem hiding this comment.
❌ Factory Review: REVERT
Verdict: REVERT
Reason: QA: ISSUES_FOUND — Feature works correctly (642 tests pass, composite 0.9643), but test_focus_accepted_with_deep_research_mode is vacuous: calls _validate_ceo_flags instead of _validate_late_flags, providing zero coverage for the deep-research allowlist change. Test passes for any mode including disallowed ones like build.
QA Analysis
Adversarial QA Report — PR #1255
Feature: Add --focus flag to factory workflow run and allow --focus for deep-research mode
Project type: CLI
Date: 2026-08-14
Smoke Test
Command: uv run factory --help
Output: Usage help displayed correctly, CLI entry point functional.
Result: PASS
Test Plan
factory workflow run <name> --focus <topic>injects focus context into AgentNodenode_contextfactory ceo /path --mode deep-research --focus <topic>no longer errors (deep-research in allowlist)- Focus only targets AgentNode instances, not FnNode/GateNode/Study nodes
- Edge cases: empty focus, no AgentNodes, disallowed modes, argparse registration
Feature Tests
Criterion 1: --focus flag registered in argparse for workflow run
Status: VERIFIED
Command:
uv run factory workflow run -h
Output:
usage: factory workflow run [-h] [--dry-run] [--focus FOCUS]
[--from-yaml PATH]
name project_path
...
--focus FOCUS Research topic or focus query passed to agent nodes
Command:
uv run python -c "
from factory.cli import build_parser
parser = build_parser()
args = parser.parse_args(['workflow', 'run', 'build', '/tmp/test', '--focus', 'auth'])
print('focus:', getattr(args, 'focus', 'NOT FOUND'))
"
Output:
focus: auth
Criterion 2: --focus injects context into AgentNode node_context
Status: VERIFIED
Command:
# Built a workflow with mixed nodes: Study, FnNode, 2x AgentNode, GateNode
# Ran _cmd_run with focus="auth module security"
# Inspected executor.node_contextOutput:
node_context keys: ['builder', 'researcher']
researcher context: Research topic: auth module security
builder context: Research topic: auth module security
study in context: False
fn_node in context: False
gate in context: False
ALL CHECKS PASSED
Criterion 3: Focus only targets AgentNode, not FnNode/GateNode/Study
Status: VERIFIED
Command:
uv run pytest tests/test_workflow_cli.py -v -k "focus_only_targets_agent_nodes"
Output:
tests/test_workflow_cli.py::TestCmdRun::test_focus_only_targets_agent_nodes PASSED
Additional direct verification:
# Workflow with Study, FnNode, 2x AgentNode, GateNode
# After focus injection:
# node_context contains ONLY: 'builder', 'researcher'
# Does NOT contain: 'study', 'fn_node', 'gate'Criterion 4: deep-research mode accepts --focus in _validate_late_flags
Status: VERIFIED
Command:
from factory.cli._ceo_helpers import _validate_late_flags
from pathlib import Path
result = _validate_late_flags(
mode='deep-research', focus='test topic',
prompt_file=None, research_ideation=None,
design_existing=False, project_path=Path('/tmp'),
no_github=False, issue_number=None, just_plan=False,
)
print(f'deep-research + focus: {result}') # None = acceptedOutput:
deep-research + focus: None (None = accepted)
Criterion 5: Disallowed modes still reject --focus
Status: VERIFIED
Command:
# Tested: build, design (no existing project), meta, founder, plan
# All returned error code 1Output:
build + focus rejected: PASS
design + focus rejected: PASS
meta + focus rejected: PASS
founder + focus rejected: PASS
plan + focus rejected: PASS
Error message verified:
Error: --focus (targeted mode) only works in improve, research, create, evolve, study,
frontend-design, frontend-design-discover, deep-research, or design (with --just-plan) mode,
got 'build'. The project must already be built before targeting specific items.
Error message correctly lists deep-research in the allowed modes.
Criterion 6: --focus with --dry-run on workflow run works end-to-end
Status: VERIFIED
Command:
uv run factory workflow run improve /tmp/test-project --focus "auth module" --dry-run
Output:
{
"workflow": "improve",
"success": true,
"halted": false,
"halt_reason": "",
"nodes_executed": 19,
"duration_ms": 14.2,
"files_produced": [...]
}Edge Case Tests
Edge 1: --focus without argument
Status: VERIFIED (correct error)
Command:
uv run factory workflow run improve /tmp/test-project --focus
Output:
factory workflow run: error: argument --focus: expected one argument
Edge 2: --focus "" (empty string)
Status: VERIFIED (no-op, no crash)
Empty string is falsy in Python, so the if focus: guard prevents injection. Workflow runs successfully without injecting empty context.
Edge 3: --focus on workflow with no AgentNodes
Status: VERIFIED (graceful no-op)
Command: Created a workflow with only Study + FnNode, ran with --focus "topic".
Output: Return code 0, node_context remains empty. No crash.
Code Reviewer Flag Investigation
Issue: test_focus_accepted_with_deep_research_mode calls _validate_ceo_flags but the allowlist is in _validate_late_flags.
Status: NOT_VERIFIED (test is vacuous)
Evidence:
# _validate_ceo_flags does NOT check the focus mode allowlist.
# It only checks mutual exclusions (focus+from_plan, focus+refine, etc.)
# The mode allowlist check is ONLY in _validate_late_flags (line 457).
#
# Proof: _validate_ceo_flags also passes for mode='build' + focus='test',
# which should be rejected. The test passes for ALL modes, not just deep-research.Impact: The test provides no actual coverage for the deep-research allowlist change. The feature works correctly (verified directly above via _validate_late_flags), but the test is misleading — it claims to verify deep-research acceptance but would pass even if deep-research were not in the allowlist.
Recommendation: The test should call _validate_late_flags instead of _validate_ceo_flags to actually verify the allowlist. This is a test quality issue, not a functional bug.
All Tests Pass
Command:
uv run pytest tests/test_workflow_cli.py tests/test_study.py -v --tb=short
Output:
175 passed in 8.11s
Acceptance Criteria Verification
| # | Criterion | Status |
|---|---|---|
| 1 | --focus flag registered for workflow run |
VERIFIED |
| 2 | Focus injects into AgentNode node_context |
VERIFIED |
| 3 | Focus skips non-AgentNode types | VERIFIED |
| 4 | deep-research mode accepts --focus |
VERIFIED |
| 5 | Disallowed modes still reject --focus |
VERIFIED |
| 6 | End-to-end dry-run works | VERIFIED |
| 7 | Test test_focus_accepted_with_deep_research_mode tests the right function |
NOT_VERIFIED |
Adversarial Verdict: FAIL
Reason: The test test_focus_accepted_with_deep_research_mode in tests/test_study.py:1504 is vacuous — it tests _validate_ceo_flags which does not contain the focus mode allowlist. The function under test would accept ANY mode with focus (including modes that should be rejected like build), so the test provides zero coverage for the actual allowlist change in _validate_late_flags. The feature itself works correctly, but the test is misleading and provides false confidence.
Posted by Factory CEO
1 participant
Summary
--focusargument tofactory workflow runCLI — injects the focus query into allAgentNodenode contexts viaexecutor.node_contextdeep-researchto the--focusallowlist infactory ceoflag validationEnables:
Used by the ReportBench evaluation harness to pipe benchmark prompts through the deep-research workflow.
Test plan
factory workflow run --helpshows--focusflagfactory workflow run deep-research /path --focus "test"passes focus to agent nodesfactory ceo /path --mode deep-research --focus "test"no longer errors