test: add coverage for venv injection in runners and eval runner

[crqu]

Summary

Adds 8 tests covering the uncovered venv injection code paths from #1368, bringing codecov/patch coverage to ~100% on both gap files (above the 79% target that blocked #1368).

• factory/runners/claude.py — covers build_command() and build_interactive_command() venv injection
• factory/eval/runner.py — covers _run_project_eval() and _run_single_project_dimension() venv injection

Tests added (in tests/test_worktree_venv.py)

Test Class	Covers	Tests
TestBuildCommandVenvEnv	ClaudeRunner.build_command()	positive + negative
TestBuildInteractiveCommandVenvEnv	ClaudeRunner.build_interactive_command()	positive + negative
TestRunProjectEvalVenvEnv	_run_project_eval()	positive + negative
TestRunSingleDimensionVenvEnv	_run_single_project_dimension()	positive + negative

Each site gets a positive test (.factory-managed marker exists → VIRTUAL_ENV and PATH injected) and a negative test (no marker → VIRTUAL_ENV absent).

Test plan

• All 19 venv tests pass (11 existing + 8 new) in 0.09s
• Full suite: 4949 passed
• Lint clean (ruff)

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.90%. Comparing base (887f317) to head (939f627).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1378      +/-   ##
==========================================
+ Coverage   84.83%   84.90%   +0.06%     
==========================================
  Files         205      205              
  Lines       23287    23287              
  Branches     3700     3700              
==========================================
+ Hits        19755    19771      +16     
+ Misses       2696     2684      -12     
+ Partials      836      832       -4     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[crqu]

@ceo-review

[github-actions]

✅ Factory Review: KEEP

Verdict: KEEP
Reason: QA: CLEAN — 5493 tests pass, composite 0.963, 7/7 code review categories PASS, 0 issues found, adversarial testing verified all 19 tests individually with 680 related tests regression-free

QA Analysis

Adversarial QA — PR #1378

PR: test: add coverage for venv injection in runners and eval runner
Detected project type: Library (Python CLI with test suite)
Date: 2026-08-21

Scope

PR #1378 adds tests/test_worktree_venv.py (288 lines, 19 tests across 7 test classes) covering per-worktree venv isolation introduced in PR #1368. The production code changes add venv injection to 5 code paths: _setup_worktree_venv, _resolve_python, _run_cmd, ClaudeRunner.build_command, ClaudeRunner.build_interactive_command, _run_project_eval, and _run_single_project_dimension.

No strategist plan file (.factory/strategy/current.md) exists; acceptance criteria derived from the PR commit and code diff.

Smoke Test

Status: VERIFIED

$ factory --help
usage: factory [-h] <command> ...
Remote Factory — domain-agnostic multi-agent software evolution loop
...

CLI entry point works. Proceeding to feature tests.

Acceptance Criteria

AC1: All 19 new tests pass

Status: VERIFIED

$ uv run pytest tests/test_worktree_venv.py -v

tests/test_worktree_venv.py::TestSetupWorktreeVenv::test_creates_venv_via_uv_sync PASSED
tests/test_worktree_venv.py::TestSetupWorktreeVenv::test_skips_when_no_pyproject_toml PASSED
tests/test_worktree_venv.py::TestSetupWorktreeVenv::test_fallback_on_uv_sync_failure PASSED
tests/test_worktree_venv.py::TestSetupWorktreeVenv::test_graceful_degradation_on_total_failure PASSED
tests/test_worktree_venv.py::TestSetupWorktreeVenv::test_fallback_venv_created_then_pip_fails PASSED
tests/test_worktree_venv.py::TestResolvePython::test_prefers_venv_python PASSED
tests/test_worktree_venv.py::TestResolvePython::test_falls_back_to_sys_executable PASSED
tests/test_worktree_venv.py::TestResolvePython::test_ignores_non_factory_venv PASSED
tests/test_worktree_venv.py::TestRunCmdVenvEnv::test_sets_venv_env_when_factory_venv_exists PASSED
tests/test_worktree_venv.py::TestRunCmdVenvEnv::test_no_venv_env_when_no_venv PASSED
tests/test_worktree_venv.py::TestRunCmdVenvEnv::test_no_venv_env_when_non_factory_venv PASSED
tests/test_worktree_venv.py::TestBuildCommandVenvEnv::test_injects_venv_when_marker_exists PASSED
tests/test_worktree_venv.py::TestBuildCommandVenvEnv::test_no_venv_without_marker PASSED
tests/test_worktree_venv.py::TestBuildInteractiveCommandVenvEnv::test_injects_venv_when_marker_exists PASSED
tests/test_worktree_venv.py::TestBuildInteractiveCommandVenvEnv::test_no_venv_without_marker PASSED
tests/test_worktree_venv.py::TestRunProjectEvalVenvEnv::test_injects_venv_when_marker_exists PASSED
tests/test_worktree_venv.py::TestRunProjectEvalVenvEnv::test_no_venv_without_marker PASSED
tests/test_worktree_venv.py::TestRunSingleDimensionVenvEnv::test_injects_venv_when_marker_exists PASSED
tests/test_worktree_venv.py::TestRunSingleDimensionVenvEnv::test_no_venv_without_marker PASSED

============================== 19 passed in 0.13s ==============================

AC2: Each test class passes in isolation (no inter-test dependencies)

Status: VERIFIED

Each of the 7 test classes was run independently:

$ uv run pytest tests/test_worktree_venv.py::TestSetupWorktreeVenv -v
5 passed in 0.07s

$ uv run pytest tests/test_worktree_venv.py::TestResolvePython -v
3 passed in 0.06s

$ uv run pytest tests/test_worktree_venv.py::TestRunCmdVenvEnv -v
3 passed in 0.07s

$ uv run pytest tests/test_worktree_venv.py::TestBuildCommandVenvEnv -v
2 passed in 0.06s

$ uv run pytest tests/test_worktree_venv.py::TestBuildInteractiveCommandVenvEnv -v
2 passed in 0.06s

$ uv run pytest tests/test_worktree_venv.py::TestRunProjectEvalVenvEnv -v
2 passed in 0.06s

$ uv run pytest tests/test_worktree_venv.py::TestRunSingleDimensionVenvEnv -v
2 passed in 0.09s

AC3: All tested functions are importable from their declared modules

Status: VERIFIED

$ uv run python -c "
from factory.worktree import WORKTREE_VENV_MARKER, _setup_worktree_venv
from factory.eval.languages.python import _resolve_python
from factory.eval.languages.base import _run_cmd
from factory.eval.runner import _run_project_eval, _run_single_project_dimension
from factory.runners.claude import ClaudeRunner
from factory.models import AgentRunRequest, ProjectEvalDimension
print('WORKTREE_VENV_MARKER =', repr(WORKTREE_VENV_MARKER))
print('All imports OK')
"

WORKTREE_VENV_MARKER = '.factory-managed'
All imports OK

AC4: is_factory_venv helper works correctly (used by all injection paths)

Status: VERIFIED

$ uv run python -c "
from pathlib import Path
import tempfile
from factory.worktree import is_factory_venv, WORKTREE_VENV_MARKER

with tempfile.TemporaryDirectory() as td:
    p = Path(td)
    assert not is_factory_venv(p), 'Should be False when no .venv exists'
    (p / '.venv').mkdir()
    assert not is_factory_venv(p), 'Should be False without marker'
    (p / '.venv' / WORKTREE_VENV_MARKER).touch()
    assert is_factory_venv(p), 'Should be True with marker'
    print('is_factory_venv: all direct assertions passed')
"

is_factory_venv: all direct assertions passed

AC5: Tests pass with warnings-as-errors (no deprecation warnings)

Status: VERIFIED

$ uv run pytest tests/test_worktree_venv.py -v -W error

============================== 19 passed in 0.11s ==============================

AC6: No regressions in related test areas

Status: VERIFIED

$ uv run pytest tests/ -v -k "worktree or venv or runner or eval" --timeout=60

==== 680 passed, 1 skipped, 4586 deselected, 3 xpassed in 165.82s (0:02:45) ====

680 tests passed across worktree, runner, and eval test areas with zero failures.

Edge Case Tests

Edge 1: Marker-based discrimination — non-factory venvs are ignored

Status: VERIFIED

Three test classes explicitly verify this: TestResolvePython::test_ignores_non_factory_venv, TestRunCmdVenvEnv::test_no_venv_env_when_non_factory_venv, and TestBuildCommandVenvEnv::test_no_venv_without_marker. A .venv/ directory without the .factory-managed marker file is correctly ignored in all code paths.

Edge 2: Fallback chain — uv sync failure falls through to venv + pip

Status: VERIFIED

TestSetupWorktreeVenv::test_fallback_on_uv_sync_failure verifies the three-step fallback (uv sync fails → python -m venv → uv pip install -e) succeeds and marks the venv.

Edge 3: Total failure — graceful degradation when all venv creation attempts fail

Status: VERIFIED

TestSetupWorktreeVenv::test_graceful_degradation_on_total_failure and test_fallback_venv_created_then_pip_fails verify that when both the primary and fallback paths fail, _setup_worktree_venv returns None without raising.

Edge 4: Async test paths (eval runner) work correctly

Status: VERIFIED

TestRunProjectEvalVenvEnv and TestRunSingleDimensionVenvEnv use AsyncMock + pytest-asyncio (auto mode) to verify the async _run_project_eval and _run_single_project_dimension functions inject venv env vars correctly.

Coverage Assessment

The 19 tests cover all 5 venv injection points in the production code:

Code Path	Tests	Positive	Negative
_setup_worktree_venv (worktree.py)	5	2 (uv sync, fallback)	3 (no pyproject, total fail, pip fail)
_resolve_python (python.py)	3	1 (prefers venv)	2 (fallback, ignores non-factory)
_run_cmd (base.py)	3	1 (sets env)	2 (no venv, non-factory venv)
build_command (claude.py)	2	1 (injects venv)	1 (no marker)
build_interactive_command (claude.py)	2	1 (injects venv)	1 (no marker)
_run_project_eval (runner.py)	2	1 (injects venv)	1 (no marker)
_run_single_project_dimension (runner.py)	2	1 (injects venv)	1 (no marker)

Process Cleanup

No servers started. No tmux sessions created. No orphaned processes.

Adversarial Verdict: PASS

All 19 new tests pass individually and as a suite. Every injection point has both positive (marker present → venv injected) and negative (no marker → no injection) coverage. The fallback chain in _setup_worktree_venv has thorough failure-mode tests. No regressions detected in the 680 related tests. No warnings. All tested functions are importable and the WORKTREE_VENV_MARKER constant resolves correctly.

---

Posted by Factory CEO