Skip to content

ci: bound apt and wget operations - #130

Merged
skilledwolf merged 1 commit into
ALPSim:masterfrom
skilledwolf:WIP/ci-network-timeouts
Aug 26, 2026
Merged

ci: bound apt and wget operations#130
skilledwolf merged 1 commit into
ALPSim:masterfrom
skilledwolf:WIP/ci-network-timeouts

Conversation

@skilledwolf

@skilledwolf skilledwolf commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Problem addressed

Occasionally CI jobs fail due to network timeouts (on apt update and wget). When it happens the CI runs to its maximum of 6 hours before the job gets killed, hung on that step. This PR adds timeout and retry guards, such that these jobs can succeed or fail within minutes with a clear error message.

Summary

  • configure apt with three retries and 30-second HTTP/HTTPS socket timeouts
  • bound the LLVM and Boost downloads with explicit wget retry and timeout options
  • cap dependency-install steps and overall Linux/macOS jobs so network stalls fail promptly
  • apply the same apt protection to the wheel-build dependency step

Validation

  • actionlint -ignore shellcheck .github/workflows/build.yml .github/workflows/build_wheels.yml
  • git diff --check
  • audited every apt update and wget call in both workflows for bounded network behavior

@skilledwolf skilledwolf self-assigned this Aug 20, 2026
@skilledwolf
skilledwolf marked this pull request as ready for review August 20, 2026 21:30
@skilledwolf

Copy link
Copy Markdown
Collaborator Author

@marcusr2ML @Ooolab Do you have time to take a look at this? the changes are very contained and near-trivial -- just some simple timeout guards

@Ooolab

Ooolab commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

Independent verification

I dug through CI history to confirm the problem and tested the fix in an Ubuntu 24.04 container plus against the real endpoints. The problem is real, and the solution works. Details below, plus two optional suggestions.

Problem confirmed from CI history

  • On 2026-08-19, at least three build.yml runs had dozens of Ubuntu jobs killed at exactly 6 hours (e.g. run 32230441285: jobs started 08:01, cancelled 14:02; runs 32272214827 / 32272211305 the same at 15:48 → 21:49).
  • The hung step is precisely Install dependencies. The raw log of one hung job (95998889195) shows apt-get update printing its last line (Get:5 https://archive.ubuntu.com ... noble-security InRelease) at 08:02:02 and then nothing for 6 hours until the runner killed it at 14:01:34 — exactly the failure mode described.

Solution tested

  • actionlint + YAML parse pass on both files; the only findings (two duplicate matrix entries) already exist on master.
  • The apt conf snippet is valid and applies: the exact printf line produces a file apt-config dump picks up (Acquire::Retries "3", http/https::Timeout "30"), and normal apt-get update + package installs are unaffected.
  • Wedged-mirror simulation (server accepts connections but never sends a byte, plus a mid-download-stall variant): with the conf in place, apt-get update gives up in ~2–4 minutes (retries × 30 s per file). One observation: stock Ubuntu 24.04 apt was also bounded in these synthetic tests (~1 attempt × 120 s default timeout), yet real CI still hung 6 h — so the production stall is evidently a mode apt's read-timer doesn't bound (e.g. a trickling connection resetting the timer). That means the hard guarantee comes from the step-level timeout-minutes: 20 / job-level caps, which this PR also adds. The layered design is the right call.
  • wget flags behave as claimed: against the stall server, the PR's flag set fails cleanly (exit 4) in ≈ tries × timeout, while default wget (900 s read timeout × 20 tries) was still hanging when I capped the test. The exact PR commands succeed against the real endpoints: llvm.sh (8.5 KB, valid bash) and the Boost 1.91 tarball (244 MB, gzip-verified).
  • Bonus correctness fix: run steps use GitHub's default bash -e without pipefail, so the old wget -qO- https://apt.llvm.org/llvm.sh | sudo bash silently succeeds when wget fails (bash reads EOF → exit 0) and the job breaks confusingly later. The new download-then-run form fails the step immediately.
  • Timeout caps have safe headroom: max successful job duration across six recent runs is 41 min on Linux (cap 90) and 53 min on macOS (cap 120) — ≥2× margin, so the caps won't kill healthy jobs.

Two optional suggestions

  1. apt-get update exits 0 when a mirror fails (warning only), so the step proceeds with stale lists rather than failing with a clear error; any real breakage only surfaces at the later apt install. Adding -o APT::Update::Error-Mode=any to the apt-get update calls would make mirror failure hard-fail at the point of the error.
  2. build_wheels.yml gains a step timeout but no job-level timeout-minutes — a hang elsewhere in the job (e.g. inside cibuildwheel) could still burn the 6-hour default. A job-level cap analogous to the ones added in build.yml would close that gap.

Neither suggestion blocks this PR — as-is it already converts the observed 6-hour hangs into ≤20-minute failures.

🤖 Generated with Claude Code

@Ooolab Ooolab left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had the timeout problem previously. Glad to see it addressed by this PR!
The suggestions are just for other potential causes of the similar problem. We can address them in the future in another PR if they do happen.

@skilledwolf
skilledwolf merged commit c2febd5 into ALPSim:master Aug 26, 2026
39 checks passed
@skilledwolf
skilledwolf deleted the WIP/ci-network-timeouts branch August 26, 2026 19:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants