fix(ci): nuke all docker state on self-hosted runners - #6049
Conversation
The project-name-filtered cleanup was a no-op: the Makefile names each Compose project <BLD_NAME>-<GITVERSION> (e.g. ubuntu22-tap-3.0.11-502-gd8bdd5a), not the bare distribution names in CI_BUILD_COMPOSE_PROJECTS, so leaked build containers survived into the next job and kept writing root-owned files into the bind-mounted checkout. Remove the dead env var and stop/take down ALL containers, volumes and networks on self-hosted runners, both preflight and post-job.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
📜 Recent review details⏰ Context from checks skipped due to timeout. (2)
🔇 Additional comments (1)
📝 WalkthroughWalkthroughThe CI workflow removes project-specific Docker Compose cleanup. It now deletes all Docker containers, volumes, and networks before eligible self-hosted builds and after jobs. Post-job cleanup runs for every job outcome on trusted runners. ChangesCI Docker cleanup
Estimated code review effort: 3 (Moderate) | ~15 minutes Mergeability Score: ⚪ Minimal · up to This is a localized CI workflow cleanup change with no actionable merge-blocking risk identified; it is merge-ready after normal checks. Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Code Review ✅ ApprovedRemoves all Docker containers, volumes, and networks on self-hosted runners at preflight and post-job to prevent workspace contamination from leaked Compose stacks. No issues found. OptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4408aae64e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| containers="$(docker ps -aq)" | ||
| if [ -n "${containers}" ]; then | ||
| docker rm -f ${containers} |
There was a problem hiding this comment.
On any self-hosted VM whose Docker daemon contains a persistent service or state from another workflow, docker ps -aq selects those containers too and docker rm -f forcibly destroys them; the following volume/network prunes can then delete their unused state. This runs both before cache-miss builds and again under always(), despite the checked-in cleanup design explicitly requiring unrelated runner workloads to remain untouched and prohibiting broad Docker pruning (docs/superpowers/specs/2026-08-11-ci-compose-workspace-cleanup-design.md:15-17,27-36), so the cleanup needs to identify only this workflow's Compose resources.
Useful? React with 👍 / 👎.
|
1 participant
Problem
Self-hosted runners reuse the working directory serially across CI-builds matrix legs. A cancelled dockerized build leaks its Compose stack, which keeps writing root-owned files into the bind-mounted checkout (
./:/opt/proxysql) while the next job starts.The existing cleanup ("Stop stale Docker Compose project") was a no-op:
CI_BUILD_COMPOSE_PROJECTS: 'debian12 ubuntu22 ubuntu24', filtering on labelcom.docker.compose.project=<bare-dist-name>.<BLD_NAME>-<GITVERSION>(e.g.ubuntu22-tap-3.0.11-502-gd8bdd5a) — seeMakefile:519. The bare distribution name is the IMG_NAME, not the Compose project name.So the leaked writer container was never matched,
chown -Rsucceeded but the live container re-created root-owned files ~1.6s later, and the contamination guard aborted the job. This produced the recurring "workspace still contains files not owned by the runner" failures.Change
Nuke all docker state on self-hosted runners at both cleanup points:
always().Both steps now remove all containers (
docker rm -fon everything running), all volumes (docker volume prune -af), and all networks (docker network prune -af). No name guessing, no filter — nothing dockerized survives into the next job.Also removed the now-dead
CI_BUILD_COMPOSE_PROJECTSenv var.Notes
Summary by cubic
Wipes all Docker state on self-hosted runners before checkout and after each job to stop root-owned file contamination from leaked Compose stacks. Old behavior: filtered cleanup by bare distribution name and missed projects named <BLD_NAME>-; containers survived into the next job. New behavior: unconditional removal of all containers, volumes, and networks on trusted self-hosted runners.
always()), gated by trusted and self-hosted runner checks.docker rm -f $(docker ps -aq)), prunes all volumes and networks; images are not pruned.CI_BUILD_COMPOSE_PROJECTS.Written for commit 4408aae. Summary will update on new commits.
Summary by CodeRabbit