fix(ci): quiesce stale self-hosted Compose projects - #6040
Conversation
Document the CI-builds failure observed on ci-vm-1: a stale Docker Compose process continues writing root-owned files into the persistent bind-mounted checkout after the ownership guard begins. Define the matrix distribution as the only permitted Compose cleanup scope, avoiding broad Docker pruning and unrelated runner workloads. Record the required preflight and postflight order, preserve ownership verification, and limit failure artifacts to build logs after a real Build-step failure. These criteria make the upcoming GH-Actions change reviewable and protect against repeating the checkout and artifact races.
Add an executable plan for repairing the self-hosted CI contamination race. It scopes Docker Compose teardown to the matrix distribution, retains ownership verification, and narrows failure artifacts to the build logs. The plan deliberately avoids a source-text test because it cannot exercise the self-hosted Docker daemon or persistent workspace. Local YAML parsing and whitespace checks provide configuration validation; the behavioral verification is a live CI-builds run after the GH-Actions change merges.
Cancelled CI-builds matrix legs can leave a Docker Compose container writing root-owned files through the persistent checkout bind mount. The next job then races the ownership guard before checkout; ci-vm-1 demonstrably ran cancelled ubuntu22 and debian12 legs immediately before the failed ubuntu24 GenAI leg. Tear down only the explicit CI-build projects (debian12, ubuntu22, and ubuntu24) before the preflight repair and again before the final repair. Keep the existing ownership verification, but ensure its chown no longer races a live build container. Limit failure artifacts to ci_build_log and require that the Build step actually failed, preventing a preflight error from trying to zip the mutable checkout. The YAML parse and diff checks pass locally; the Docker lifecycle requires a live self-hosted CI rerun after merge for behavioral verification.
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
🚧 Files skipped from review as they are similar to previous changes (3)
📜 Recent review details⏰ Context from checks skipped due to timeout. (2)
📝 WalkthroughWalkthroughSelf-hosted CI now cleans configured Docker Compose projects before checkout and after job completion. Cleanup failures remain fatal. Failure artifacts are limited to ChangesCompose workspace cleanup
Estimated code review effort: 3 (Moderate) | ~20 minutes Sequence Diagram(s)sequenceDiagram
participant SelfHostedRunner
participant DockerCompose
participant Workspace
participant Build
participant ArtifactUploader
SelfHostedRunner->>DockerCompose: Clean configured projects before checkout
DockerCompose->>Workspace: Remove stale containers and volumes
SelfHostedRunner->>Build: Run Build and Check build
SelfHostedRunner->>DockerCompose: Clean configured projects after the job
Build->>ArtifactUploader: Report Build or Check build failure
ArtifactUploader->>Workspace: Read proxysql/ci_build_log/
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5b5c7ad790
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| workspace="${GITHUB_WORKSPACE:?GITHUB_WORKSPACE is not set}" | ||
| compose_file="${workspace}/proxysql/docker-compose.yml" | ||
|
|
||
| if [ -f "${compose_file}" ]; then |
There was a problem hiding this comment.
On a self-hosted runner whose previous checkout cleanup partially removed proxysql/ before hitting an ownership error, docker-compose.yml may already be gone while the old containers remain alive. This guard then silently skips every teardown, allowing the stale bind-mount writer to race the following chown exactly as before. Discover and remove these explicitly named projects through their Docker Compose project labels, or otherwise make missing configuration a handled cleanup case rather than treating it as success.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci-builds.yml:
- Around line 165-173: Make both cleanup loops in
.github/workflows/ci-builds.yml at lines 165-173 and 668-676 continue after any
docker-compose down failure by recording a failure status, then exit non-zero
only after all three compose projects have been attempted. Update
docs/superpowers/plans/2026-08-11-ci-compose-workspace-cleanup.md at lines 49-58
to document this exhaustive cleanup and deferred failure behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 423d2eaf-18be-40ff-8a88-0d2ea7693ce0
📒 Files selected for processing (3)
.github/workflows/ci-builds.ymldocs/superpowers/plans/2026-08-11-ci-compose-workspace-cleanup.mddocs/superpowers/specs/2026-08-11-ci-compose-workspace-cleanup-design.md
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
- GitHub Check: Gitar
🧰 Additional context used
🧠 Learnings (2)
📚 Learning: 2026-04-11T13:17:55.508Z
Learnt from: renecannao
Repo: sysown/proxysql PR: 5607
File: doc/GH-Actions/README.md:13-18
Timestamp: 2026-04-11T13:17:55.508Z
Learning: When using GitHub-flavored Markdown headings, be aware that an em-dash surrounded by spaces (written as ` — `) affects the generated anchor/slug: GitHub replaces spaces with hyphens and removes non-alphanumeric punctuation, which can produce double hyphens (e.g., `## Foo — bar` → anchor `#foo--bar`, not `#foo-bar`). If you reference these anchors (e.g., internal links), ensure the expected slug matches this behavior.
Applied to files:
docs/superpowers/plans/2026-08-11-ci-compose-workspace-cleanup.mddocs/superpowers/specs/2026-08-11-ci-compose-workspace-cleanup-design.md
📚 Learning: 2026-04-11T13:17:55.509Z
Learnt from: renecannao
Repo: sysown/proxysql PR: 5607
File: doc/GH-Actions/README.md:13-18
Timestamp: 2026-04-11T13:17:55.509Z
Learning: When reviewing GitHub-flavored Markdown links/anchors, remember that heading-to-anchor slug generation treats spaces as hyphens and removes punctuation. If a heading contains an em-dash surrounded by spaces (e.g. ` — `), the slugs can legitimately include a double hyphen where the two surrounding space-runs become `-` on either side of the removed em-dash (e.g. `...vocabulary--read...`). Do not flag double-hyphens in anchor links for em-dash-containing headings as errors; they reflect GitHub’s correct slug behavior.
Applied to files:
docs/superpowers/plans/2026-08-11-ci-compose-workspace-cleanup.mddocs/superpowers/specs/2026-08-11-ci-compose-workspace-cleanup-design.md
🪛 LanguageTool
docs/superpowers/plans/2026-08-11-ci-compose-workspace-cleanup.md
[uncategorized] ~14-~14: The official name of this software platform is spelled with a capital “H”.
Context: ...modify v3.0 or PR #6038. - Touch only .github/workflows/ci-builds.yml for the implem...
(GITHUB)
docs/superpowers/specs/2026-08-11-ci-compose-workspace-cleanup-design.md
[uncategorized] ~15-~15: The official name of this software platform is spelled with a capital “H”.
Context: ...d with ENOENT. ## Scope Modify only .github/workflows/ci-builds.yml on a branch ba...
(GITHUB)
🔇 Additional comments (3)
docs/superpowers/specs/2026-08-11-ci-compose-workspace-cleanup-design.md (1)
1-53: LGTM!docs/superpowers/plans/2026-08-11-ci-compose-workspace-cleanup.md (1)
1-48: LGTM!Also applies to: 64-95
.github/workflows/ci-builds.yml (1)
155-164: LGTM!Also applies to: 174-177, 660-667, 679-680, 701-710
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
There was a problem hiding this comment.
1 issue found across 3 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name=".github/workflows/ci-builds.yml">
<violation number="1" location=".github/workflows/ci-builds.yml:172">
P3: The new preflight and postflight Compose cleanup steps are near-identical copies: same shell block, same path derivation, same `-p debian12/ubuntu22/ubuntu24` loop, and the same `down -v --remove-orphans`. If the project set or the Docker command ever changes (e.g. adding a distribution or switching to `docker compose`), it has to be edited in two places and can silently drift. Worth factoring the teardown into a small callable action or shared snippet so the project list is defined once.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
| if [ -f "${compose_file}" ]; then | ||
| for compose_project in debian12 ubuntu22 ubuntu24; do | ||
| echo ">>> stopping stale Compose project: ${compose_project}" | ||
| docker-compose -f "${compose_file}" -p "${compose_project}" down -v --remove-orphans |
There was a problem hiding this comment.
P3: The new preflight and postflight Compose cleanup steps are near-identical copies: same shell block, same path derivation, same -p debian12/ubuntu22/ubuntu24 loop, and the same down -v --remove-orphans. If the project set or the Docker command ever changes (e.g. adding a distribution or switching to docker compose), it has to be edited in two places and can silently drift. Worth factoring the teardown into a small callable action or shared snippet so the project list is defined once.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At .github/workflows/ci-builds.yml, line 172:
<comment>The new preflight and postflight Compose cleanup steps are near-identical copies: same shell block, same path derivation, same `-p debian12/ubuntu22/ubuntu24` loop, and the same `down -v --remove-orphans`. If the project set or the Docker command ever changes (e.g. adding a distribution or switching to `docker compose`), it has to be edited in two places and can silently drift. Worth factoring the teardown into a small callable action or shared snippet so the project list is defined once.</comment>
<file context>
@@ -152,10 +152,29 @@ jobs:
+ if [ -f "${compose_file}" ]; then
+ for compose_project in debian12 ubuntu22 ubuntu24; do
+ echo ">>> stopping stale Compose project: ${compose_project}"
+ docker-compose -f "${compose_file}" -p "${compose_project}" down -v --remove-orphans
+ done
+ fi
</file context>
Address the review findings in the self-hosted cleanup path. Define the CI-build Compose project set once, attempt every project even when one docker-compose teardown fails, and remove explicitly labelled containers when docker-compose.yml was already lost during a partial cleanup. The latter closes the exact pre-checkout gap where a stale bind-mount writer could otherwise survive. Give Check build an id and gate the log artifact with failure() plus either Build or Check build failure. This preserves the preflight exclusion while making the artifact run after a failed build under GitHub Actions status semantics. Update the design and plan with the fallback and deferred-error contract, including the corrected documentation typo.
Code Review ✅ ApprovedStops stale Docker Compose projects and refines artifact collection in self-hosted CI workflows to prevent workspace contamination and race conditions. No issues found. OptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change the behavior for this request:
Was this helpful? React with 👍 / 👎 | Gitar |
|
The prior #6038 CI-builds run used the cleanup from #6040, which invoked the unavailable standalone docker-compose executable. Its Debian build completed but failed in post-build cleanup, and the Ubuntu 24 leg failed during pre-build cleanup. PR #6041 is now merged into GH-Actions and switches both cleanup loops to Docker Compose v2. This empty commit changes no ProxySQL source; it triggers #6038 again so the corrected reusable workflow is exercised on the self-hosted runners.
1 participant
Summary
debian12,ubuntu22, andubuntu24) before the pre-checkout ownership guard and again before final workspace cleanupci_build_logwhen the Build step itself fails, avoiding a secondary artifact-zipping failure for preflight workspace errorsRoot cause
On PR #6038,
ci-vm-1ran cancelledubuntu22anddebian12CI-builds legs immediately before the failedubuntu24,-tap-genai-gcovleg. The ownership guard ranchown -R, then immediately found new root-owned Connector/C objects, proving a stale Compose container was still writing to the shared workspace. The fallback artifact upload subsequently raced the same mutable tree and failed withENOENT.The cleanup is constrained to the three Compose project names used by the active CI-builds matrix; it does not use a broad Docker prune and does not change workflow permissions.
Validation
.github/workflows/ci-builds.ymland.github/workflows/ci-trigger.ymlwith PyYAML.git diff --check origin/GH-Actions...HEAD.A live self-hosted CI-builds run after merge is required to verify the Docker lifecycle behavior.
Summary by cubic
Prevents stale Docker Compose containers on self-hosted runners from writing into the persistent checkout and breaking later CI legs. We stop the CI projects (
debian12,ubuntu22,ubuntu24) before ownership repair and again at teardown, and only upload build logs when Build or its log check fails.CI_BUILD_COMPOSE_PROJECTSonce and attempt every project; stop viadocker-compose ... down -v --remove-orphans, with a label-based container fallback whendocker-compose.ymlis missing.proxysql/ci_build_log/whenbuildorcheck_buildfailed; useif-no-files-found: warn.Written for commit 1323a8f. Summary will update on new commits.
Summary by CodeRabbit
Bug Fixes
Documentation