Fix E2E chat test timeout: move E2E tests to local Docker pre-push workflow, add WebGPU/Vulkan support - #68
Conversation
…2E_DTYPE Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com> Agent-Logs-Url: https://github.com/devlux76/q2/sessions/3ac97559-4d4e-424f-9282-36e2a2e2f589
|
@copilot I agree with what you said here, "chat.spec.ts and model-loading.spec.ts timed out in CI because onnx-community/Qwen3.5-0.8B-ONNX (~400 MB q4) is too large to download and run on WASM within the 8-minute window on a CPU-only ubuntu-latest runner — the only available backend in that environment." What do we need to change to get a GPU runner in there, otherwise we're going to spend a lot of money on CPU time when when we could spend a lot less on GPU time and not be bottle necked here. We can't be the only ones testing WebGPU apps E2E in a pipeline like this. I'll bet theres even a specific runner for this that would save us some time on our setup not just testing. |
…, three-tier CI docs Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com> Agent-Logs-Url: https://github.com/devlux76/q2/sessions/0dbd6e2c-0b47-4820-b874-7734ab896901
Done in commit ci.yml — runs-on: ${{ vars.E2E_RUNNER || 'ubuntu-latest' }}Set Three tiers are documented in the workflow comments:
playwright.config.ts — ...( process.env.E2E_GPU_AVAILABLE ? [] : ['--use-gl=swiftshader'] ),On CPU runners it forces Mesa software GL (always works). On GPU runners (
|
|
@copilot As it turns out this was sort of a mistake. GitHub does not offer GPU instance instances on Pro as far as I can tell. It seems these are only available on Enterpriise which I cannot afford. What we can do is to create a docker container and run it locally and make it part of the precommit hook. This docket container should have Vulkan since my own system is Intel/AMD. I'll just have it run locally before committing. So thank you for your work, but we need to move e2e back to the laptop. A docker container with vulkan and playwright is likely the best solution. I'd go with a self hosted runner, but after investigating further this cannot be made secure. What can be made secure is local e2e testing before pushing as part of the bun run build, bun run test locally. Thank you for your hardwork. Please just get this part right (remove e2e in CI and place it locally) and I think we're complete on the entire issue. |
Co-authored-by: devlux76 <86517969+devlux76@users.noreply.github.com> Agent-Logs-Url: https://github.com/devlux76/q2/sessions/cf642d47-b83d-4618-a569-3d7f3c7a70bf
Done in commit What changed:
To run manually: |
There was a problem hiding this comment.
Pull request overview
This PR moves Playwright E2E coverage out of GitHub Actions (where CPU-only runners time out on large models) and into a local Docker-based workflow that can use WebGPU/Vulkan when available, while also adding env-driven model dtype overrides for the E2E specs.
Changes:
- Removed the
e2ejob from CI and documented the new local E2E execution path. - Added a Docker-based local E2E runner (
e2e/Dockerfile,e2e/run-local.sh) and wired it intopackage.jsonplus a.husky/pre-pushhook. - Updated E2E specs to support
E2E_DTYPEoverrides and updated Playwright Chromium launch args to support WebGPU.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| playwright.config.ts | Adds Chromium launch args intended to enable WebGPU and select software vs hardware paths. |
| package.json | Adds test:e2e:local script as the entry point for Docker-based E2E runs. |
| e2e/run-local.sh | New Docker orchestration script for local Playwright runs with optional /dev/dri passthrough. |
| e2e/model-loading.spec.ts | Adds E2E_DTYPE support for selecting model dtype during the loading test. |
| e2e/chat.spec.ts | Adds E2E_DTYPE support for selecting model dtype during the chat test. |
| e2e/Dockerfile | New local E2E image (Playwright base + Mesa Vulkan drivers + Bun). |
| .husky/pre-push | Runs the local Docker E2E suite before git push (skips when Docker is absent). |
| .github/workflows/ci.yml | Removes CI E2E job and adds guidance for running E2E locally. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # ── Run tests ───────────────────────────────────────────────────────────── | ||
| echo "▶ Running E2E tests..." | ||
| exec docker run --rm \ | ||
| --ipc=host \ | ||
| ${GPU_FLAGS[@]+"${GPU_FLAGS[@]}"} \ | ||
| -v "$REPO_ROOT:/app" \ | ||
| -v "$NM_VOLUME:/app/node_modules" \ | ||
| -w /app \ | ||
| -e CI=true \ | ||
| -e E2E_GPU_AVAILABLE="${E2E_GPU_AVAILABLE:-$GPU_AVAILABLE}" \ | ||
| -e E2E_MODEL="${E2E_MODEL:-}" \ | ||
| -e E2E_DTYPE="${E2E_DTYPE:-}" \ |
There was a problem hiding this comment.
run-local.sh always injects E2E_MODEL/E2E_DTYPE into the container, even when they’re unset on the host (they become empty strings). Because the specs use nullish coalescing (??), an empty string prevents the defaults from being applied and selectOption/model id filling can fail. Only pass these -e flags when the vars are non-empty, or pass explicit defaults (e.g., omit the env vars unless set).
| # ── Run tests ───────────────────────────────────────────────────────────── | |
| echo "▶ Running E2E tests..." | |
| exec docker run --rm \ | |
| --ipc=host \ | |
| ${GPU_FLAGS[@]+"${GPU_FLAGS[@]}"} \ | |
| -v "$REPO_ROOT:/app" \ | |
| -v "$NM_VOLUME:/app/node_modules" \ | |
| -w /app \ | |
| -e CI=true \ | |
| -e E2E_GPU_AVAILABLE="${E2E_GPU_AVAILABLE:-$GPU_AVAILABLE}" \ | |
| -e E2E_MODEL="${E2E_MODEL:-}" \ | |
| -e E2E_DTYPE="${E2E_DTYPE:-}" \ | |
| # ── Optional E2E env flags ──────────────────────────────────────────────── | |
| E2E_FLAGS=() | |
| if [ -n "${E2E_MODEL:-}" ]; then | |
| E2E_FLAGS+=(-e "E2E_MODEL=$E2E_MODEL") | |
| fi | |
| if [ -n "${E2E_DTYPE:-}" ]; then | |
| E2E_FLAGS+=(-e "E2E_DTYPE=$E2E_DTYPE") | |
| fi | |
| # ── Run tests ───────────────────────────────────────────────────────────── | |
| echo "▶ Running E2E tests..." | |
| exec docker run --rm \ | |
| --ipc=host \ | |
| ${GPU_FLAGS[@]+"${GPU_FLAGS[@]}"} \ | |
| ${E2E_FLAGS[@]+"${E2E_FLAGS[@]}"} \ | |
| -v "$REPO_ROOT:/app" \ | |
| -v "$NM_VOLUME:/app/node_modules" \ | |
| -w /app \ | |
| -e CI=true \ | |
| -e E2E_GPU_AVAILABLE="${E2E_GPU_AVAILABLE:-$GPU_AVAILABLE}" \ |
| const MODEL_ID = process.env.E2E_MODEL ?? 'onnx-community/Qwen3.5-0.8B-ONNX'; | ||
| const MODEL_DTYPE = process.env.E2E_DTYPE ?? 'q4'; |
There was a problem hiding this comment.
process.env.E2E_MODEL ?? ... / process.env.E2E_DTYPE ?? ... will treat an empty string as a valid override. With the new Docker runner currently passing empty env vars, this can result in MODEL_ID/MODEL_DTYPE being '' and the test failing. Prefer || (or a small normalization helper) so empty strings fall back to defaults.
| const MODEL_ID = process.env.E2E_MODEL ?? 'onnx-community/Qwen3.5-0.8B-ONNX'; | |
| const MODEL_DTYPE = process.env.E2E_DTYPE ?? 'q4'; | |
| const MODEL_ID = process.env.E2E_MODEL || 'onnx-community/Qwen3.5-0.8B-ONNX'; | |
| const MODEL_DTYPE = process.env.E2E_DTYPE || 'q4'; |
| const MODEL_ID = process.env.E2E_MODEL ?? 'onnx-community/Qwen3.5-0.8B-ONNX'; | ||
| const MODEL_DTYPE = process.env.E2E_DTYPE ?? 'q4'; |
There was a problem hiding this comment.
Same issue as in chat.spec.ts: using ?? here will not fall back when E2E_MODEL/E2E_DTYPE are present but empty (which the new Docker runner currently does). Use || (or normalize env vars) so empty values don’t break model selection.
| const MODEL_ID = process.env.E2E_MODEL ?? 'onnx-community/Qwen3.5-0.8B-ONNX'; | |
| const MODEL_DTYPE = process.env.E2E_DTYPE ?? 'q4'; | |
| const MODEL_ID = process.env.E2E_MODEL || 'onnx-community/Qwen3.5-0.8B-ONNX'; | |
| const MODEL_DTYPE = process.env.E2E_DTYPE || 'q4'; |
| * | ||
| * The default model is onnx-community/Qwen3.5-0.8B-ONNX (q4). Override with | ||
| * the E2E_MODEL environment variable for faster CI runs. | ||
| * the E2E_MODEL and E2E_DTYPE environment variables for faster CI runs. |
There was a problem hiding this comment.
The header comment still says the env vars are for “faster CI runs”, but this PR removes E2E from CI. Consider updating the wording to “faster local runs” (or similar) to avoid misleading guidance.
| * the E2E_MODEL and E2E_DTYPE environment variables for faster CI runs. | |
| * the E2E_MODEL and E2E_DTYPE environment variables for faster local runs. |
| # | ||
| # Update this tag when you bump @playwright/test in package.json. | ||
| # Available tags: https://mcr.microsoft.com/v2/playwright/tags/list | ||
| FROM mcr.microsoft.com/playwright:v1.50.1-jammy |
There was a problem hiding this comment.
The Playwright base image version (v1.50.1-jammy) is far behind the repo’s @playwright/test version (currently ^1.58.2). Version mismatches between the runner and the preinstalled browsers/drivers can cause flaky or broken E2E runs. Update the image tag to match the Playwright version used by the repo (and consider pinning playwright/@playwright/test consistently).
| FROM mcr.microsoft.com/playwright:v1.50.1-jammy | |
| FROM mcr.microsoft.com/playwright:v1.58.2-jammy |
| * Without GPU (lavapipe fallback): | ||
| * --use-gl=swiftshader forces the Mesa software Vulkan rasterizer. | ||
| * WebGPU still works; inference falls back to WASM. |
There was a problem hiding this comment.
The comment claims --use-gl=swiftshader “forces the Mesa software Vulkan rasterizer (lavapipe)”, but SwiftShader is Chromium’s software GL backend and is distinct from Mesa’s Vulkan lavapipe. This is likely to confuse future debugging of WebGPU/Vulkan issues; please adjust the comment to describe the actual fallback(s) being used.
chat.spec.tsandmodel-loading.spec.tstimed out in CI becauseonnx-community/Qwen3.5-0.8B-ONNX(~400 MB q4) is too large to download and run on WASM within the 8-minute window on a CPU-onlyubuntu-latestrunner — the only available backend in that environment. GitHub Actions Pro does not offer GPU-enabled runners, so E2E tests are moved to a local Docker pre-push workflow instead of CI.Local Docker E2E runner
A new
e2e/Dockerfilebuilds on the official Playwright Jammy image and layers Mesa Vulkan drivers on top for Intel/AMD GPU support:/dev/dripassthrough (hardware WebGPU)e2e/run-local.shorchestrates the run:/dev/driand passes it to the container for real GPU access; falls back to lavapipe automaticallynode_modulesin a named Docker volume (q2-e2e-node-modules) for fast subsequent runsE2E_GPU_AVAILABLE=1only when/dev/driis actually present, soplaywright.config.tsuses hardware GL vs. SwiftShader correctlyPre-push git hook
.husky/pre-pushrunsbun run test:e2e:localautomatically before everygit push. It skips gracefully if Docker is not installed and can be bypassed withgit push --no-verify.package.jsonscriptA new
test:e2e:localscript (bash e2e/run-local.sh) is the single entry point for running E2E tests locally.CI change
The
e2ejob is removed from.github/workflows/ci.yml. A comment block explains where E2E tests now live and how to run them.Conditional SwiftShader / hardware GL
--use-gl=swiftshaderis included only whenE2E_GPU_AVAILABLEis unset, so systems with a real GPU use hardware Vulkan/ANGLE and systems without fall back to Mesa software rendering:E2E_DTYPEenv varBoth
chat.spec.tsandmodel-loading.spec.tsreadprocess.env.E2E_DTYPE ?? 'q4'for the dtype selection, parallel to the existingE2E_MODELoverride.Original prompt
📱 Kick off Copilot coding agent tasks wherever you are with GitHub Mobile, available on iOS and Android.