@W-23385030@ feat(e2e): run Playwright specs against the Code Builder image -

[jonnyhork]

@W-23385030@
This adds an e2e harness that runs our Playwright specs against the real Code Builder (Agentforce Vibes) image, so we can catch bugs in the desktop extension build before it publishes to the marketplace instead of testing that path by hand.

The key idea: Code Builder runs a Node extension host over a browser UI, so it's the desktop build, not the web build. We drive a browser at the container's code-server URL, and swap the unreleased monorepo VSIX into the running container at runtime — no changes needed in code-builder-images. Like the desktop e2e leaves, we test the artifacts the release is about to ship, not a rebuild: the workflow downloads the VSIX from an upstream Build All run by runId. A filesystem version gate then asserts every in-scope extension resolves to exactly one dir at that version before any spec runs, so we can't false-green against stale code. The design and the rejected alternatives are written up in ADR 0022.

Whats in here:

• playwright-vscode-ext: createContainerConfig + createContainerTest (container config factory and fixture, reusing the existing Electron-decoupled page objects)
• salesforcedx-vscode-core: a test:container wireit script, container fixtures, two specs (configList.container.spec.ts for the CLI-shellout path, seededWorkspace.container.spec.ts for the mounted fixture), a checked-in fixture project under test/playwright/fixtures/container-workspace/, and its config
• .github/workflows/codeBuilderE2E.yml plus the swap, version-gate, and workspace-seed scripts
• scripts/codeBuilderLocalE2E.ts — a one-command local twin of the workflow (npm run test:container:local), so you can run the whole pull → swap → restart → gate → specs loop against your working tree without CI
• ADR 0022 and doc updates

Its manual-only for now (workflow_dispatch/workflow_call), so merging this runs nothing automatically — it just registers the dispatch button on develop. To dispatch it you pass the runId of a recent Build All run (the one whose VSIX you want to test) and, if the artifact isnt the default VS Code Extensions label, the artifactName. It slots straight into the e2e.yml fan-out later, since that already threads a build runId. I kept the spec set small on purpose — one CLI-shellout spec and one that proves the seeded fixture opens — to prove the concept; the team can decide which specs to port from there.

The scripts (codeBuilderLocalE2E.ts, codeBuilderSwapExtensions.ts, codeBuilderVerifyExtensions.ts, codeBuilderSeedWorkspace.ts) are ts-node, matching the repo convention, and use execFileSync with arg arrays rather than shell interpolation so there's no shell-injection surface on the docker/gh/sf calls.

Heads up that Im expecting the first manual dispatch to be a shakeout — the container mechanics (swap, restart, org re-auth) are the parts weve only run by hand so far. Merging registers the workflow so I can dispatch it against develop and iterate.

How CI is allowed to pull the image

The Code Builder image (ghcr.io/forcedotcom/code-builder-images/workspace-manager/codebuilder) is a private GHCR package in the same forcedotcom org as this repo, so the pull needs authentication — but no App, PAT, or private key lives in the workflow.

The job requests packages: read permission, then logs in to ghcr.io with the run's built-in GITHUB_TOKEN (the ephemeral token GitHub mints per run, -u ${{ github.actor }}). That token is allowed to read the package because the codebuilder package grants Manage Actions access to salesforcedx-vscode — a package-level setting on the code-builder-images side that lets this repo's Actions token pull it. So the chain is: package grants this repo Actions read → the per-run GITHUB_TOKEN inherits that read → docker login + docker pull succeed. Nothing to rotate, nothing stored as a secret.

That grant is scoped to the repo's Actions token only — it does not authorize a human pulling from their laptop. But the same repo-permissions model covers local: the team has read on the image repo and the codebuilder package inherits repo permissions, so each dev pulls as their own GitHub user (below) — no shared bot user or PAT.

Dev-hub auth is unrelated to the image and reuses the existing SFDX_AUTH_URL_E2E secret.

Running it locally

npm run test:container:local runs the same pull → swap → restart → gate → specs pipeline against your working tree. What you need:

• Docker running.
• gh (GitHub CLI) logged in to github.com. The script pulls the image as your own GitHub user — gh auth token for the credential, gh api user for the username — so no shared token and nothing to rotate. The one catch is scope: ghcr requires the read:packages scope, and a default gh auth login doesnt request it, so if the login fails the script prints the one-line fix (gh auth refresh -h github.com -s read:packages). Escape hatch: set CR_PAT to your own classic PAT (read:packages, SSO-authorized for forcedotcom) and the script uses that instead of gh.
• sf logged in to a dev hub (the script reuses minimalTestOrg if present, else creates it). If sf isnt installed globally it falls back to npx @salesforce/cli.
• --run-id additionally needs gh (to download the CI artifact) — already covered unless youre on the CR_PAT path.

The preflight assumes a fresh box — it checks docker (+ a running daemon) and gh (installed + logged in) up front and, if anything's missing, prints one consolidated report with copy-paste fixes rather than failing partway through a pull. Provenance for whatever it ends up testing (upstream run metadata for --run-id, or your git branch + HEAD for a local build) is logged at the top of the run.

Known limitation: the swap trusts semver, and semver can lie

Worth calling out before merge, because it shapes where this goes next. The swap and the version gate currently lean on the extension semver, and there are two ways that bites us.

First, how the swap takes at all. The image bakes a curated extension set into /base/extension-overrides, and on boot its start.sh symlinks each override into the runtime dir only when the override is a strictly-newer semver than whats already linked. So a same-or-lower VSIX would never re-link. We work around that by rm -rf-ing both the baked override dir and the runtime symlink before installing ours, so start.shtreats ours as new. That works, but it rides two contracts thecode-builder-images` team owns and can change under us: the strictly-newer relink rule, and semver moving forward.

Second, and the sharper one: the gate compares installed semver against the artifact's semver, not the bytes. Extension versions arent bumped until release, so last nights unreleased build and the marketplace build can both read 67.4.0. If the swap ever silently no-ops (glob miss, a path the image renamed, a permissions hiccup), the host keeps the baked production copy, the gate compares 67.4.0 == 67.4.0, and the run goes green while testing production. Thats exactly the false-green the gate was meant to stop. The OK ...@67.4.0 lines prove the dirs exist at that version — they dont prove the bytes are the build we meant to test.

Options to close this, cheapest first (details + trade-offs in ADR 0022):

• Content check — hash the VSIX bytes on the host before swap, hash the installed override dir after restart, assert equal. Byte-exact proof; smallest change; closes the false-green today.
• Build stamp — bake a git SHA / build timestamp into the VSIX at package time, gate on that instead of/alongside semver, and surface it in the provenance banner. Makes the version genuinely unique and doubles as provenance. Needs a packaging change.
• Unconditional wipe — instead of clearing in-scope dirs to win the relink race, delete every salesforce.* extension from the runtime + override locations unconditionally, then install ours as the only copy present. Removes the dependency on semver precedence entirely — with nothing to compare against, ours is the only thing that can load. Trade-off: tests a clean install rather than an upgrade-over-existing, and it should wipe by publisher glob rather than the hardcoded ID list so a baked extension under an unlisted ID cant survive.

These arent mutually exclusive: wipe proves only ours can load; a content/stamp check proves the bytes are the ones intended. The likely direction is wipe + a content check, which makes correctness independent of how the image team handles versioning. Official pre-release versions from code-builder-images (a future intent) would satisfy todays semver gate — but by fixing the version contract rather than removing our dependence on it, so its a nice-to-have on top, not a substitute. Filing follow-up work; this PR keeps the semver gate and documents the gap.

Seeding the workspace with metadata

The container now opens a real project instead of a bare one, so specs can open a class, run a test, or deploy. A version-controlled SFDX project lives at packages/salesforcedx-vscode-core/test/playwright/fixtures/container-workspace/ (sfdx-project.json + a force-app Apex class and its test to start; grow it as specs need). The docker run bind-mounts it into the container, and scripts/codeBuilderSeedWorkspace.ts writes coder.json via docker exec so code-server opens it — same docker exec lever we already use to disable workspace trust and swap extensions. Its wired into both CI and the local loop, with one seeded spec (seededWorkspace.container.spec.ts) that opens the fixture Apex class from the Explorer to prove the mount reached the editor.

This is the fixture-project approach Shane flagged — the closest prior art was test-workspaces/sfdx-workspace/ (a checked-in SFDX fixture, but consumed in place by the LWC/Aura language-server unit tests, and LWC-only with no Apex), so I modeled a small container-specific fixture on it rather than reusing that one.

Why a mount and not the image's own seeding: what opens in the workbench is normally decided by the image. On first boot it runs sfdx-setup.sh (once — gated behind a ~/.codebuilder marker, so it does not re-run on our swap docker restart); given SFDX_COBU_PROJECTNAME it sfdx project generates a bare project and writes coder.json. Theres also a dormant SFDX_COBU_GITHUB_PROJECT_URL git-clone branch (nothing sends it, untouched since 2023). We deliberately avoid both: they fire only on first boot behind the .codebuilder gate and live in image code the CB team owns and can change, whereas a mount is applied by Docker at run time, survives restart, and depends on nothing inside the image except the coder.json folder query. Same decoupling principle as the swap options above.

Two container-path details worth knowing: the mount targets /home/codebuilder/fixture-project, not the SFDX_COBU_PROJECTNAME path (~/e2e-project) — mounting over that would collide with sfdx project generate --output-dir /home/codebuilder, which aborts on a non-empty dir. And if you change which folder opens, the disable-workspace-trust step and the workbench-ready wait have to stay consistent (the trust setting is workspace-agnostic, so it already covers the mount). Details in ADR 0022.

Creating metadata mid-spec via createApexClass / createAndDeployApexTestClass (the desktop pattern) is still available for specs that want to exercise the create path, but the mounted fixture is the default since it gives a fixed starting state with no image involvement.

Order of operations

Heres what one dispatched CI run does end to end, for anyone reading this cold:

flowchart TD
    A["Download VSIX under test from the Build All runId (the bytes about to ship)"] --> B["docker login ghcr.io with the repo GITHUB_TOKEN"]
    B --> C["docker pull the private Code Builder image"]
    C --> D["Auth dev hub (SFDX_AUTH_URL_E2E) + create scratch org"]
    D --> E["docker run: code-server on 58080 to 8123, bind-mount fixture project, org auth from scratch-org token"]
    E --> F["Wait for the workbench to answer at localhost:8123"]
    F --> F2["Disable workspace trust (else Restricted Mode, extensions never activate)"]
    F2 --> F3["Seed: write coder.json so code-server opens the mounted fixture project"]
    F3 --> G["Swap: remove baked override dirs, install downloaded VSIX into extension-overrides"]
    G --> H["docker restart: extension host reloads new VSIX AND org re-auths"]
    H --> I{"Version gate: exactly one dir per extension at the shipping version?"}
    I -->|no| X["Fail the run — wrong or mixed versions"]
    I -->|yes| J["npm run test:container: Playwright drives the browser at the container"]
    J --> K["Always: upload report + container logs, tear down container, delete scratch org"]

Loading

Same thing as a numbered list if the diagram doesnt render for you:

1. Download the VSIX under test from the upstream Build All run (runId) — these are the exact bytes about to ship, not a rebuild.
   The run logs the artifact's provenance here — the source run's workflow, branch, commit, and build timestamp — because the VSIX semver isnt release-bumped and so also matches the marketplace build. The semver alone cant tell you whether youre testing shipping or pre-release bytes; the provenance banner (and the artifact SHA256 the download step prints) is the source of truth for what's under test.
2. docker login ghcr.io with the repo GITHUB_TOKEN and pull the private Code Builder image.
3. Auth the dev hub and create a scratch org; export its instance URL + access token.
4. docker run the image (code-server 58080 published to 8123), bind-mounting the fixture project to /home/codebuilder/fixture-project; the container auths the org from the scratch-org token on start.
5. Wait for the workbench to answer at localhost:8123.
6. Disable workspace trust in the container — otherwise the workspace opens in Restricted Mode and the Salesforce extensions never activate (the desktop fixture uses --disable-workspace-trust; this is the code-server equivalent).
7. Seed the workspace: write coder.json via docker exec so code-server opens the mounted fixture project (real metadata) instead of the image's bare generated one.
8. Swap: remove the baked (published-version) override dirs, install the downloaded VSIX into /base/extension-overrides.
9. docker restart — the extension host reloads holding the new VSIX, org auth re-runs, and the workbench comes back opening the seeded fixture.
10. Version gate: assert exactly one override dir per in-scope extension, at the shipping version. Fail loud otherwise.
11. npm run test:container — Playwright drives a browser against the running container.
12. Always: upload the HTML report + container logs, tear down the container, delete the scratch org.