E2E smoke (L4a) CI: high failure rate and long execution time

[Million-mo]

E2E smoke (L4a) CI: high failure rate and long execution time

Problem

The E2E smoke (L4a) CI job has a high failure rate and long execution time, blocking PR merges frequently.

Evidence

Recent CI runs (last 30 runs on refactor/rename-to-wolfharness branch)

Run ID	Conclusion	Duration
31175177394	❌ failure	15m44s
31174489667	🚫 cancelled	21m39s
31173163323	🚫 cancelled	21m39s
31167763855	❌ failure	10m54s
31167421314	❌ failure	10m45s
31164973604	❌ failure	11m2s
31163884596	❌ failure	10m25s
31163680060	❌ failure	—
31176469933	✅ success	11m35s
31176114429	✅ success	14m38s

Even successful runs take 11–15 minutes, close to the 20-minute timeout.

Failure patterns from run 31175177394

4 failed, 145 passed, 17 skipped, 7039 deselected, 3 xfailed, 3 xpassed in 834.44s (0:13:54)

Failed tests:

1. tests/e2e/test_acp_config.py::test_session_set_model — Timeout (>60.0s) from pytest-timeout
2. tests/e2e/test_opencode_provider_mode_config.py::test_get_mode[subprocess_server0] — httpx.ReadTimeout
3. tests/e2e/test_opencode_provider_mode_config.py::test_get_config[subprocess_server0] — httpx.ReadTimeout
4. tests/e2e/test_opencode_provider_mode_config.py::test_patch_config[subprocess_server0] — httpx.ReadTimeout

Throughout the run, hundreds of ERROR Background task failed: acp.exceptions.RequestError: Internal error messages were logged, suggesting server instability during the test session.

Root cause hypotheses

1. Subprocess server startup/teardown races: Multiple tests spawn wolfharness serve-* subprocesses. The server cache helps reuse, but crash detection and health checks may not be robust enough under CI resource constraints.

2. Timeout too aggressive for CI: The 60s per-test timeout (--timeout=60) may be too tight for subprocess-heavy tests on GitHub Actions runners (shared CPU, cold caches). The overall job timeout is 20 minutes, and with 149 tests running for ~14 minutes, there is little headroom.

3. Background task cleanup issues: The repeated acp.exceptions.RequestError: Internal error from background tasks suggests async cleanup is not completing properly between tests, causing cascading failures.

4. Resource exhaustion on CI runners: E2e tests spawn real subprocess servers. Under CI (2-core runners), concurrent subprocess management may exhaust file descriptors or ports.

Proposed investigation areas

• Increase per-test timeout from 60s → 120s for e2e tests, or make it configurable via env var
• Investigate acp.exceptions.RequestError: Internal error root cause in background task cleanup
• Audit subprocess server cache health check logic — ensure stale/crashed servers are detected and restarted
• Consider splitting E2e tests into parallel jobs (by protocol: ACP, OpenCode, AG-UI) to reduce wall time
• Add retry logic for httpx.ReadTimeout in OpenCode provider mode config tests
• Profile E2e test suite to identify the slowest 5 tests and optimize or mark as @pytest.mark.slow

Current CI config

E2E smoke (L4a):
  runs-on: ubuntu-latest
  timeout-minutes: 20
  steps:
    - run: uv run pytest -m "e2e and not slow" --timeout=60 --tb=short --no-cov -q

149 tests, ~14 minutes average, 20-minute hard timeout.

Impact

• PR merge blocked: E2e failures block all PRs, even when the failure is flaky/environmental
• Developer productivity: 15-minute feedback loop for CI is too slow for iterative development
• False negatives: Tests that pass locally fail on CI due to resource constraints