Skip to content

feat(hip): migrate every ROCm consumer to 10.0.0 from digest-pinned images (ADR-1225) - #1386

Draft
lusoris wants to merge 2 commits into
masterfrom
feat/rocm-10-migration
Draft

feat(hip): migrate every ROCm consumer to 10.0.0 from digest-pinned images (ADR-1225)#1386
lusoris wants to merge 2 commits into
masterfrom
feat/rocm-10-migration

Conversation

@lusoris

@lusoris lusoris commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Summary

AMD froze the repo.radeon.com/rocm/apt/ channel at 7.2.4 when ROCm moved to
the "TheRock" build/release system at 7.14 — apt/7.14 and apt/10.0.0 both 404,
the manylinux channel stops at rocm-rel-7.2.4, and the TheRock wheel index carries
only 7.14.0 alphas. The fork was heading into 1.0.0 on a ROCm-legacy toolchain with
no version-string edit available to get off it.

This takes ROCm from the digest-pinned rocm/dev-ubuntu-24.04:10.0.0-full image
instead, across every consumer at once: the dev container, both published GPU
images, and both CI HIP lanes.

Three things ROCm 10 changed that this had to handle — each found by a check
that failed, not by reading release notes:

  1. /opt/rocm/{bin,lib,include,llvm,…} are /etc/alternatives symlinks into
    /opt/rocm/core-10.0/. Copying or extracting /opt/rocm alone lands a directory
    of dangling links; both the rocm-src stage and the extractor repoint them at
    core-10.0/<name> relative to /opt/rocm.
  2. libamdhip64.so's link closure widened to 19 libraries / 234 MB
    (libLLVM, libclang-cpp, libamd_comgr, librocm_kpack,
    librocprofiler-register, the rocm_sysdeps bundle), wired by
    $ORIGIN-relative RPATHs. The node-rocm stage's flat two-library copy no
    longer loads; it now ships the whole closure with its layout intact.
  3. gfx1036 is natively supported, so HSA_OVERRIDE_GFX_VERSION=10.3.0 is dropped
    from dev/docker-compose.yml — keeping it would alias the agent to gfx1030
    while meson compiles gfx1036 code objects.

docker pull is not viable in CI (8.2 GB compressed / 29 GB extracted, on a runner
that already carries CUDA and oneAPI), so scripts/ci/install-rocm-from-image.sh
streams each layer blob from the registry straight into tar, extracting only
/opt/rocm minus the math libraries libvmaf never links. Peak disk is the 5.5 GB
result, not the 29 GB image.

The prune list (19 GB → 5.5 GB) is gated by a hipcc smoke compile in the same
stage. That gate earned its keep: an earlier pass removed librocprofiler-register
with a tidy-looking librocprof* glob, and every HIP binary then failed at load
with cannot open shared object file.

Verification

Measured on the dev host (Linux 7.2.3-1-cachyos, AMD gfx1036), not inferred:

rocminfo                      → Agent 2  Name: gfx1036
amdgpu-arch                   → gfx1036            (native, no HSA override)
hipcc --offload-arch=gfx1036  → compiles; kernel dispatches (h[63] = 63)
libvmaf build                 → 1537/1537 targets
meson test (HIP suite)        → Ok: 19, Expected Fail: 4, Fail: 0
end-to-end                    → hip vmaf = 45.315104 / cpu vmaf = 45.315104

Container stage, built and run:

rocm-src prune                 → 19 GB → 5.5 GB, hipcc still emits a runnable binary
COPY into an ubuntu:26.04 stage → hipcc gfx1036 OK, no dangling top-level symlinks
node-rocm copy set             → ldd libamdhip64.so: 0 unresolved, 397 MB
extractor (registry → tarball) → 5.5 GB, hipcc gfx1036 compiles and the binary runs
extractor (multi-arch index)   → resolves linux/amd64, clean error on a non-ROCm image

Reproducer / smoke-test command

# Extract ROCm 10 from the pinned image on any Linux host with curl + jq + tar
scripts/ci/install-rocm-from-image.sh \
  --image rocm/dev-ubuntu-24.04@sha256:a90cf047f615abe70fbef83c64def0a2d549ef37a39c8ea545430aba4981b374 \
  --dest /tmp/rocm10

printf '#include <hip/hip_runtime.h>\n#include <cstdio>\n__global__ void k(int*o){o[threadIdx.x]=threadIdx.x;}\nint main(){printf("ok\\n");return 0;}\n' > /tmp/smoke.hip
ROCM_PATH=/tmp/rocm10 LD_LIBRARY_PATH=/tmp/rocm10/lib \
  /tmp/rocm10/bin/hipcc --offload-arch=gfx1036 /tmp/smoke.hip -o /tmp/smoke && /tmp/smoke

# Dev container, end to end
docker compose --project-directory "$(git rev-parse --show-toplevel)" \
  -f dev/docker-compose.yml build dev-mcp
docker exec vmaf-dev-mcp vmaf \
  --reference /workspace/python/test/resource/yuv/src01_hrc00_576x324.yuv \
  --distorted /workspace/python/test/resource/yuv/src01_hrc01_576x324.yuv \
  --width 576 --height 324 --pixel_format 420 --bitdepth 8 --backend hip

Deep-dive deliverables (ADR-0108)

  • Research digestdocs/research/1225-rocm-10-distribution-and-layout.md: the channel survey, the alternatives layout, the link closure, the prune measurements, and why CI streams instead of pulling.
  • Decision matrixADR-1225 ## Alternatives considered (six options, including the deferred GHCR mirror).
  • AGENTS.md invariant notedev/AGENTS.md (never restore the apt path; never prune librocprofiler-register; HSA_OVERRIDE_GFX_VERSION stays out) and the root AGENTS.md container-pin section.
  • Reproducer / smoke-test command — above.
  • Changelog fragmentchangelog.d/changed/1225-rocm-10-therock-migration.md, CHANGELOG.md regenerated.
  • Rebase notedocs/rebase-notes.md § feat/rocm-10-migration (ADR-1225), five invariants.

Docs (rule 10)

State (rule 13)

no state delta: dependency migration — no bug opened, closed, or ruled not-affecting.

ffmpeg-patches (rule 14)

no patch impact: no C-API surface, CLI flag, meson_options.txt entry or public
header changes; the meson edit is comment-only.

🤖 Generated with Claude Code

@lusoris
lusoris force-pushed the feat/rocm-10-migration branch from 3f11190 to 20bdf88 Compare September 7, 2026 10:14
Lusoris and others added 2 commits September 7, 2026 14:16
…mages

AMD froze the repo.radeon.com apt channel at 7.2.4 when ROCm moved to the
"TheRock" build/release system at 7.14: apt/7.14 and apt/10.0.0 both 404,
the manylinux channel stops at rocm-rel-7.2.4, and the TheRock wheel index
carries only 7.14.0 alphas. The fork was therefore stuck on a ROCm-legacy
toolchain heading into 1.0.0, with no version-string edit available to get
off it.

Take ROCm from the digest-pinned rocm/dev-ubuntu-24.04:10.0.0-full image
instead. dev/Containerfile, docker/Dockerfile.production-gpu and
docker/Dockerfile.node copy a pruned /opt/rocm out of it; the two CI HIP
lanes use the new scripts/ci/install-rocm-from-image.sh, which streams the
image's /opt/rocm out of the registry layer by layer. A docker pull is not
viable on a runner that already carries CUDA and oneAPI -- 8.2 GB
compressed, 29 GB extracted -- so nothing is ever stored whole: peak disk
is the 5.5 GB result.

Three things ROCm 10 changed that this had to handle:

  - /opt/rocm/{bin,lib,include,llvm,...} are now /etc/alternatives symlinks
    into /opt/rocm/core-10.0/. Copying or extracting /opt/rocm alone lands
    dangling links, so both the rocm-src stage and the extractor repoint
    them at core-10.0/<name> relative to /opt/rocm.
  - libamdhip64.so's link closure widened to 19 libraries (234 MB) wired by
    $ORIGIN-relative RPATHs. The node-rocm runtime stage's flat copy of
    libamdhip64.so* + libhsa-runtime64.so* no longer loads; it now ships
    the whole closure with its layout intact, verified at 0 unresolved
    libraries in a scratch stage carrying only the copied files.
  - gfx1036 is natively supported, so HSA_OVERRIDE_GFX_VERSION=10.3.0 is
    dropped from dev/docker-compose.yml. Keeping it would alias the agent
    to gfx1030 while meson compiles gfx1036 code objects.

The prune list (19 GB -> 5.5 GB) is gated by a hipcc smoke compile in the
same stage: an earlier pass removed librocprofiler-register with a tidy
librocprof* glob and every HIP binary then failed at load.

Renovate follows the image digest now rather than scraping the frozen apt
directory listing.

Verified on AMD gfx1036 under Linux 7.2.3: HIP suite 19 Ok / 0 Fail, and an
end-to-end HIP score of 45.315104, bit-identical to the CPU reference.

Refs: ADR-1225

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The badge scraped `ARG ROCM_VER=` out of dev/Containerfile, which this
branch removes along with the apt install. Read the version out of the
digest-pinned rocm/dev-ubuntu-24.04:<version>-full reference instead, so it
keeps reporting a live value rather than going blank.

Verified by running the badge's own regex against the new Containerfile:
matches `rocm/dev-ubuntu-24.04:10.0.0-full`, renders `10.0.0`.

Refs: ADR-1225

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@lusoris
lusoris force-pushed the feat/rocm-10-migration branch from 20bdf88 to a334256 Compare September 7, 2026 12:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant