Skip to content

Speed up NVHPC install in CI (parallel download, igzip, mv instead of installer copy) #101

Description

@joshkamm

While adapting install_nvhpc.sh to a Claude Code cloud environment (which enforces a ~5-minute setup budget), several techniques cut the install from an estimated 10+ minutes to ~1.5–3 minutes. They should transfer to CI's cache-miss path in .github/actions/setup-build-env/install_nvhpc.sh, and one of them (a smaller install) would also speed up cache save/restore on every run.

Numbers below were measured in the cloud container (4 vCPUs); the GPU runner's bandwidth/disk will differ, so it's worth re-measuring there before adopting.

Where the time goes on a cache miss

wget -O - <tarball> | tar xpz + install breaks down into three serial costs:

  1. Download (6.98 GB tarball): single-stream HTTP from developer.download.nvidia.com did ~16 MB/s → ~7 min. With 16 parallel HTTP range requests (curl -r start-end × 16, or aria2c -x16 -s16) the same host sustained ~120 MB/s → ~1 min.
  2. Decompression: gzip -d is single-threaded and managed only ~41 MB/s of compressed input (~3 min of pure CPU). igzip from the isal package (ISA-L) did ~168 MB/s — 4x faster. Drop-in: ... | igzip -dc | tar xp ....
  3. Installer copy: the silent installer's only heavy step is tar cf - $i | (cd $INSTALL_DIR; tar xf -) over the whole 15 GB extracted tree (install_components/install line ~303). When NVHPC_INSTALL_DIR is on the same filesystem as the extraction dir, replacing that one line with mkdir -p "$INSTALL_DIR/`dirname $i`" && mv "$i" "$INSTALL_DIR/$i" makes it instant and halves peak disk usage. Everything else the installer does (CUDA symlinks, modulefiles, cmake configs, NVHPC_INSTALL_TYPE=auto) runs unmodified. A grep guard before the sed patch makes a future SDK layout change fail loudly instead of silently falling back to the copy.

Download and extraction can also be overlapped (download fixed-size chunks in parallel, stream them in order into igzip | tar as they land), which makes the total roughly max(download, extract) instead of the sum.

Every-run win: smaller cache

Linux_x86_64/25.1/profilers (Nsight Compute/Systems) is ~2.1 GB of the ~15 GB install. If CI never profiles, trimming it shrinks the actions/cache entry, speeding up save/restore on all runs and reducing eviction pressure against the repo's 10 GB cache quota (fewer evictions = fewer slow cache-miss installs).

Tested implementation

A version combining all of the above (chunked parallel download piped through igzip | tar, guarded sed patch of the installer, trims) was tested end-to-end in the cloud environment: full install in 1m17s (2m40s on a slower run), followed by a successful pixi install build of SlaterGPU against it.

Related: #23 (nvhpc packaging), #8 (original piped wget|tar setup).

🤖 Created with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions