Skip to content

fix(ci): retry transient trigger polling failures - #6042

Merged
renecannao merged 5 commits into
GH-Actionsfrom
fix/ci-trigger-api-polling
Aug 12, 2026
Merged

fix(ci): retry transient trigger polling failures#6042
renecannao merged 5 commits into
GH-Actionsfrom
fix/ci-trigger-api-polling

Conversation

@renecannao

@renecannao renecannao commented Aug 12, 2026

Copy link
Copy Markdown
Contributor

Summary

  • replace gh run watch --exit-status with explicit JSON status polling
  • retry transient GitHub API/TLS failures for up to five consecutive minutes
  • keep confirmed CI-builds failures distinct from polling-infrastructure failures
  • preserve the existing workflow topology, permissions, downstream gates, and timeout

Root cause

gh run watch --exit-status returns a nonzero status both when the watched run fails and when polling GitHub's API fails. A transient TLS certificate error therefore marked CI-trigger as failed even though all four CI-builds matrix jobs later completed successfully. Because downstream workflows require a successful CI-trigger conclusion, the polling error caused the full test fan-out to be skipped.

Behavior

Polling calls are bounded to 30 seconds and retry after 5, 10, 20, then at most 30 seconds. A successful API response resets the five-minute consecutive-error window. A confirmed completed conclusion other than success still fails immediately and reports the exact conclusion.

Validation

  • parsed the workflow with PyYAML and verified its timeout/topology invariants
  • ShellCheck passed on the extracted trigger script
  • transient failure followed by success: passed
  • successful response resets the retry window: passed
  • missing run followed by visibility: passed
  • continuous API failure reaches the bounded infrastructure error: passed
  • confirmed build failure remains an immediate distinct failure: passed
  • git diff --check origin/GH-Actions...HEAD: passed

Hosted fan-out verification requires this change to be merged into GH-Actions, after which an empty PR commit can exercise the shared reusable workflow.


Summary by cubic

Makes CI-trigger resilient to transient GitHub API/TLS errors by replacing gh run watch --exit-status with explicit JSON polling, bounded retries, and hard per-call timeouts. Prevents skipped downstream runs by clearly separating polling failures from real CI-builds failures.

  • Bug Fixes
    • Replace watcher with gh JSON polling + jq and a retry helper (30s hard per-call timeout via timeout --signal=KILL; 5→10→20→30s backoff; five-minute consecutive-error window).
    • Distinguish infra polling errors from confirmed CI-builds conclusions; discovery every 20s, status every 30s; successful API calls reset the error window.
    • Preserve workflow topology, permissions, and 240-minute timeout; update design and plan docs for the new polling flow.

Written for commit ea66648. Summary will update on new commits.

Review in cubic

Summary by CodeRabbit

  • Bug Fixes

    • Improved CI-trigger reliability when checking build status.
    • Added bounded retries and polling to handle temporary service or API interruptions.
    • CI now clearly distinguishes successful, failed, missing, and inconclusive build results.
    • Prevented indefinite waiting by enforcing command and overall time limits.
  • Documentation

    • Added design and implementation guidance for resilient CI status monitoring and validation.

@coderabbitai

coderabbitai Bot commented Aug 12, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4a79255c-93e4-45dd-8c64-6f4aceea5af7

📥 Commits

Reviewing files that changed from the base of the PR and between af0c664 and ea66648.

📒 Files selected for processing (3)
  • .github/workflows/ci-trigger.yml
  • docs/superpowers/plans/2026-08-12-ci-trigger-api-polling.md
  • docs/superpowers/specs/2026-08-12-ci-trigger-api-polling-design.md
🚧 Files skipped from review as they are similar to previous changes (3)
  • .github/workflows/ci-trigger.yml
  • docs/superpowers/specs/2026-08-12-ci-trigger-api-polling-design.md
  • docs/superpowers/plans/2026-08-12-ci-trigger-api-polling.md

📝 Walkthrough

Walkthrough

The CI-trigger workflow now uses bounded GitHub CLI polling. It discovers runs by SHA, retries transient API failures, polls JSON status, and succeeds only after a confirmed successful conclusion. Supporting documents define testing, validation, and hosted verification.

Changes

CI-trigger polling

Layer / File(s) Summary
Polling contract and acceptance criteria
docs/superpowers/specs/..., docs/superpowers/plans/...
The design and plan define bounded commands, retry behavior, error classification, polling intervals, preserved workflow constraints, and acceptance criteria.
Workflow discovery and completion polling
.github/workflows/ci-trigger.yml, docs/superpowers/plans/...
The workflow adds gh_poll, JSON-based SHA discovery, configurable polling values, and explicit success or failure handling for completed runs.
Recovery, failure, and verification procedures
docs/superpowers/plans/...
The plan defines fake CLI scenarios, terminal failure tests, static checks, clean-state verification, branch audits, and hosted workflow checks.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant CI-trigger
  participant gh_poll
  participant GitHub CLI
  CI-trigger->>gh_poll: discover CI-builds run by candidate SHA
  gh_poll->>GitHub CLI: query run list as JSON
  GitHub CLI-->>gh_poll: matching run ID or transient error
  CI-trigger->>gh_poll: request run status
  gh_poll->>GitHub CLI: query run status and conclusion
  GitHub CLI-->>gh_poll: completed status or transient error
  gh_poll-->>CI-trigger: successful completion or failure
Loading

Possibly related PRs

Poem

I’m a rabbit watching runs in flight,
Bounded retries keep timing right.
JSON shows each build’s state,
Confirmed success opens the gate.
A carrot waits in green delight!

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: retrying transient failures during CI trigger polling.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/ci-trigger-api-polling

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In @.github/workflows/ci-trigger.yml:
- Line 60: Use a non-ignorable KILL timeout for every gh invocation: update
.github/workflows/ci-trigger.yml:60 and
docs/superpowers/plans/2026-08-12-ci-trigger-api-polling.md:124 to use
--signal=KILL, and update
docs/superpowers/specs/2026-08-12-ci-trigger-api-polling-design.md:33-34 to
state that the timeout forcibly terminates the command at its deadline.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: 4824c3ae-b06c-4fd8-aa2c-28016da7b52e

📥 Commits

Reviewing files that changed from the base of the PR and between b16ff81 and af0c664.

📒 Files selected for processing (3)
  • .github/workflows/ci-trigger.yml
  • docs/superpowers/plans/2026-08-12-ci-trigger-api-polling.md
  • docs/superpowers/specs/2026-08-12-ci-trigger-api-polling-design.md
📜 Review details
⏰ Context from checks skipped due to timeout. (1)
  • GitHub Check: Gitar
🧰 Additional context used
🧠 Learnings (3)
📚 Learning: 2026-04-11T13:17:55.508Z
Learnt from: renecannao
Repo: sysown/proxysql PR: 5607
File: doc/GH-Actions/README.md:13-18
Timestamp: 2026-04-11T13:17:55.508Z
Learning: When using GitHub-flavored Markdown headings, be aware that an em-dash surrounded by spaces (written as ` — `) affects the generated anchor/slug: GitHub replaces spaces with hyphens and removes non-alphanumeric punctuation, which can produce double hyphens (e.g., `## Foo — bar` → anchor `#foo--bar`, not `#foo-bar`). If you reference these anchors (e.g., internal links), ensure the expected slug matches this behavior.

Applied to files:

  • docs/superpowers/specs/2026-08-12-ci-trigger-api-polling-design.md
  • docs/superpowers/plans/2026-08-12-ci-trigger-api-polling.md
📚 Learning: 2026-04-11T13:17:55.509Z
Learnt from: renecannao
Repo: sysown/proxysql PR: 5607
File: doc/GH-Actions/README.md:13-18
Timestamp: 2026-04-11T13:17:55.509Z
Learning: When reviewing GitHub-flavored Markdown links/anchors, remember that heading-to-anchor slug generation treats spaces as hyphens and removes punctuation. If a heading contains an em-dash surrounded by spaces (e.g. ` — `), the slugs can legitimately include a double hyphen where the two surrounding space-runs become `-` on either side of the removed em-dash (e.g. `...vocabulary--read...`). Do not flag double-hyphens in anchor links for em-dash-containing headings as errors; they reflect GitHub’s correct slug behavior.

Applied to files:

  • docs/superpowers/specs/2026-08-12-ci-trigger-api-polling-design.md
  • docs/superpowers/plans/2026-08-12-ci-trigger-api-polling.md
📚 Learning: 2026-08-11T12:56:09.846Z
Learnt from: renecannao
Repo: sysown/proxysql PR: 6033
File: docs/superpowers/plans/2026-08-11-ed25519-authentication.md:469-469
Timestamp: 2026-08-11T12:56:09.846Z
Learning: In `docs/superpowers/plans/2026-08-11-ed25519-authentication.md`, the historical-artifact notice states that embedded expected outputs are plan-time values. Review-driven changes can modify the MariaDB Ed25519 implementation and TAP assertion counts after the plan is written. The shipped implementation and tests are authoritative, so reviewers must not require retroactive synchronization of plan-time expected outputs.

Applied to files:

  • docs/superpowers/plans/2026-08-12-ci-trigger-api-polling.md
🪛 LanguageTool
docs/superpowers/specs/2026-08-12-ci-trigger-api-polling-design.md

[uncategorized] ~10-~10: The official name of this software platform is spelled with a capital “H”.
Context: ...jected the TLS certificate returned for api.github.com; CI-trigger reported CI-builds a...

(GITHUB)


[uncategorized] ~19-~19: The official name of this software platform is spelled with a capital “H”.
Context: ...r build failure. ## Scope Modify only .github/workflows/ci-trigger.yml and its direc...

(GITHUB)

docs/superpowers/plans/2026-08-12-ci-trigger-api-polling.md

[uncategorized] ~13-~13: The official name of this software platform is spelled with a capital “H”.
Context: .... ## Global Constraints - Modify only .github/workflows/ci-trigger.yml and the direc...

(GITHUB)


[uncategorized] ~33-~33: The official name of this software platform is spelled with a capital “H”.
Context: ...** - Consumes: the existing GH_TOKEN, ${{ github.repository }}, `${{ github.event.after...

(GITHUB)


[uncategorized] ~33-~33: The official name of this software platform is spelled with a capital “H”.
Context: ...GH_TOKEN, ${{ github.repository }}, ${{ github.event.after || github.sha }}, `${{ git...

(GITHUB)


[uncategorized] ~33-~33: The official name of this software platform is spelled with a capital “H”.
Context: ...{ github.event.after || github.sha }}, ${{ github.event.pull_request.head.sha || github.s...

(GITHUB)


[uncategorized] ~33-~33: The official name of this software platform is spelled with a capital “H”.
Context: ...request.head.sha || github.sha }}, and ${{ github.sha }}values. - Produces: the sameC...

(GITHUB)


[uncategorized] ~59-~59: The official name of this software platform is spelled with a capital “H”.
Context: ...``` Extract the first run block from `.github/workflows/ci-trigger.yml` with this scr...

(GITHUB)

🔇 Additional comments (1)
.github/workflows/ci-trigger.yml (1)

11-37: LGTM!

Also applies to: 92-140

Comment thread .github/workflows/ci-trigger.yml Outdated
@renecannao
renecannao marked this pull request as ready for review August 12, 2026 08:06
@gitar-bot

gitar-bot Bot commented Aug 12, 2026

Copy link
Copy Markdown
Code Review ✅ Approved

Replaces gh run watch with explicit JSON status polling and bounded retries to prevent transient GitHub API and TLS errors from failing CI triggers. No issues found.

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing less information.

Comment with these commands to change the behavior for this request:

Auto-apply Compact
gitar auto-apply:on         
gitar display:verbose         

Was this helpful? React with 👍 / 👎 | Gitar

@renecannao
renecannao merged commit 68c561b into GH-Actions Aug 12, 2026
3 checks passed

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ea66648b95

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment on lines +103 to +105
RUNID=$(jq -r --arg sha "${expected_sha}" \
'first(.[] | select(.displayTitle | contains($sha)) | .databaseId) // empty' \
<<<"${GH_RESPONSE}")

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Prefer the newly launched build over stale same-SHA runs

When CI is retriggered for an unchanged head SHA and the new CI-builds run is not visible during the first discovery request, this first(...) can select a completed run from an earlier invocation because the query now includes every status. The local gh run list --help documents --status as “Filter runs by status”; unlike the previous -s in_progress query, this call applies no such filter. The poller then stops discovery and may either approve the trigger before the fresh build finishes or propagate the stale run's failure, so discovery should exclude stale completed runs or correlate the run with the current invocation.

Useful? React with 👍 / 👎.

@sonarqubecloud

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant