diff --git a/.github/workflows/README_BUILD_PACKAGES.md b/.github/workflows/README_BUILD_PACKAGES.md new file mode 100644 index 00000000..bbaecd83 --- /dev/null +++ b/.github/workflows/README_BUILD_PACKAGES.md @@ -0,0 +1,239 @@ +# Building Relocatable TransferBench Packages with GitHub Actions + +This document describes the GitHub Actions workflow for building relocatable +TransferBench packages using the ROCm SDK from +[TheRock](https://github.com/ROCm/TheRock). + +The workflow (`.github/workflows/build-relocatable-packages.yml`) and the +`build_packages_local.sh` script at the repo root produce: + +- **DEB** packages for Ubuntu/Debian +- **RPM** packages for AlmaLinux/Rocky/RHEL (built in `manylinux_2_28`) +- **TGZ** archives for any Linux distribution + +All packages install to `/opt/rocm/extras-` and use relocatable +`$ORIGIN`-relative `RPATH` so the install tree itself can be moved without +hard-coded library paths. These artifacts are **not** fully self-contained: +target systems must still provide the required ROCm/HSA runtime libraries +(declared as package dependencies: `hsa-rocr` and `numactl`). + +This workflow is modeled on the +[ROCmValidationSuite packaging workflow](https://github.com/ROCm/ROCmValidationSuite/blob/master/.github/workflows/README_BUILD_PACKAGES.md). + +## Workflow Triggers + +| Trigger | Behavior | +|---------|----------| +| Push to `develop`, `mainline`, `release/**` | Build + upload to S3 (if configured) + regenerate apt/yum repo metadata | +| Pull request to `develop`, `mainline` | Build + upload to ref-specific S3 path (no repo metadata) | +| Schedule (daily 13:00 UTC) | Same as push, with auto-fetched latest ROCm | +| `workflow_dispatch` | Manual trigger with `rocm_version` and `gpu_family` inputs | + +### Manual trigger inputs + +- **`rocm_version`** (e.g. `7.11.0a20260121`). Empty = auto-fetch latest from TheRock. +- **`gpu_family`** — one of: + - `gfx94X-dcgpu` (MI300A/MI300X) — **default** + - `gfx950-dcgpu` (MI350X/MI355X) + - `gfx110X-all` (RX 7900 XTX, 7800 XT, 7700S, Radeon 780M) + - `gfx120X-all` (RX 9060/XT, 9070/XT) + - `gfx1151` (Strix Halo iGPU) + +## Build features enabled in CI + +The workflow always builds with: + +- `ENABLE_NIC_EXEC=OFF` — RDMA NIC executor disabled (would require libibverbs.so.1 at runtime; not bundled by TheRock SDK) +- `ENABLE_MPI_COMM=OFF` — MPI multi-node communicator disabled (would require OpenMPI at runtime; not bundled by TheRock SDK). Packages are built to run out of the box with only `numactl`/`libnuma1` from the OS. +- `DISABLE_DMABUF=OFF` — DMA-BUF support for GPU Direct RDMA +- `BUILD_RELOCATABLE_PACKAGE=ON` — RVS-style install prefix + package naming +- `GPU_TARGETS` — full data-center + consumer set (gfx906, 908, 90a, 942, 950, 1030, 1100/01/02, 1150/51, 1200/01) + +## Local builds + +The same script the workflow uses also works locally: + +```bash +# Auto-fetch latest ROCm +sudo ./build_packages_local.sh + +# Pin a specific version (use sudo -E to preserve env) +sudo -E ROCM_VERSION=7.11.0a20260121 GPU_FAMILY=gfx94X-dcgpu ./build_packages_local.sh + +# Debug build +sudo -E BUILD_TYPE=Debug ./build_packages_local.sh +``` + +`sudo` is required because the script installs system packages +(`libnuma-dev`, `libibverbs-dev`, `libopenmpi-dev`, etc). + +After the script completes, packages live under `build/`: + +``` +build/amdrocm7-transferbench_1.66.02-_amd64.deb +build/amdrocm7-transferbench-1.66.02-.x86_64.rpm +build/amdrocm7-transferbench-1.66.02-Linux.tar.gz +``` + +## Installing built packages + +### Ubuntu / Debian + +```bash +sudo dpkg -i build/amdrocm7-transferbench_*.deb +/opt/rocm/extras-7/bin/TransferBench +``` + +### Rocky / RHEL / AlmaLinux + +```bash +sudo rpm -i --replacefiles --nodeps build/amdrocm7-transferbench-*.rpm +/opt/rocm/extras-7/bin/TransferBench +``` + +### Any Linux (TGZ — relocatable install tree, requires ROCm runtime on target) + +End-user instructions (pre-install ROCm, runtime dependencies, extract, +`PATH` / `LD_LIBRARY_PATH`, troubleshooting) live in the project docs at +[docs/install/INSTALL_TGZ.rst](../../docs/install/INSTALL_TGZ.rst). + +Quick smoke test from the repo root after a successful build: + +```bash +sudo mkdir -p /opt/rocm/extras-7 +sudo tar -xzf build/amdrocm7-transferbench-*.tar.gz -C /opt/rocm/extras-7 --strip-components=1 +export PATH=/opt/rocm/extras-7/bin:$PATH +# With no args, TransferBench prints version, usage, available presets, +# and detected topology — a fast end-to-end check that the binary loads +# its ROCm libs correctly. +TransferBench +``` + +> Once the `help` and `smoketest` presets land on `develop` (currently on +> `candidate`), prefer `TransferBench help` for usage and `TransferBench +> smoketest` for a real correctness check. + +## S3 upload (OIDC) + +S3 upload runs only when: +- The repository is `ROCm/TransferBench`, **and** +- The `AWS_S3_BUCKET` repository variable is set. + +Upload uses **AWS OIDC** — no long-term keys are stored in the repo. + +### S3 path layout + +| Trigger | Path | +|---------|------| +| `release/*` push or dispatch | `release/transferbench/{deb,rpm,tar}/` | +| Schedule, push to `develop`/`mainline`, dispatch on non-release | `nightly/transferbench/{deb,rpm,tar}/` | +| Pull request (same repo) | `transferbench///{ubuntu-22.04,manylinux_2_28}/` | + +### Required repository setup + +In **Settings → Secrets and variables → Actions**: + +**Secrets tab:** +- `AWS_ROLE_ARN` — IAM role ARN with OIDC trust for this repo (e.g. `arn:aws:iam::123456789012:role/rocm-transferbench-s3-upload`) + +**Variables tab:** +- `AWS_S3_BUCKET` — bucket name (e.g. `rocm-transferbench-packages`) +- `RUNNER_LABEL` (optional) — override Ubuntu runner label (default `ubuntu-22.04`) +- `RUNNER_LABEL_CONTAINER` (optional) — override container-job runner label (default `ubuntu-latest`) +- `RUNNER_LABEL_UTILITY` (optional) — override summary-job runner label (default `ubuntu-latest`) + +### IAM role trust policy + +The role in `AWS_ROLE_ARN` must trust GitHub's OIDC provider: + +```json +{ + "Version": "2012-10-17", + "Statement": [{ + "Effect": "Allow", + "Principal": { + "Federated": "arn:aws:iam:::oidc-provider/token.actions.githubusercontent.com" + }, + "Action": "sts:AssumeRoleWithWebIdentity", + "Condition": { + "StringEquals": { "token.actions.githubusercontent.com:aud": "sts.amazonaws.com" }, + "StringLike": { "token.actions.githubusercontent.com:sub": "repo:ROCm/TransferBench:*" } + } + }] +} +``` + +Permissions needed: `s3:PutObject`, `s3:GetObject`, `s3:ListBucket`, `s3:DeleteObject` on the bucket. + +## Using the S3 paths as apt / yum repos + +Push and scheduled builds also publish APT / YUM metadata so the S3 paths +work directly as native package repositories. + +### apt (Ubuntu / Debian) + +```bash +echo "deb [trusted=yes] https://.s3.amazonaws.com/nightly/transferbench/deb/ ./" \ + | sudo tee /etc/apt/sources.list.d/transferbench-nightly.list +sudo apt update +sudo apt install amdrocm7-transferbench +``` + +### yum / dnf (Rocky / RHEL / AlmaLinux) + +```bash +sudo tee /etc/yum.repos.d/transferbench-nightly.repo <<'EOF' +[transferbench-nightly] +name=TransferBench Nightly +baseurl=https://.s3.amazonaws.com/nightly/transferbench/rpm/ +enabled=1 +gpgcheck=0 +EOF +sudo dnf install amdrocm7-transferbench +``` + +> **Note:** `[trusted=yes]` / `gpgcheck=0` skip GPG verification. For +> production deployments, sign packages and metadata with a GPG key. + +## Verifying RPATH + +```bash +readelf -d /opt/rocm/extras-7/bin/TransferBench | grep -E 'RPATH|RUNPATH' +# Should contain $ORIGIN, $ORIGIN/../lib, /opt/rocm/extras-7/lib +``` + +## Troubleshooting + +### S3 step fails with "Credentials could not be loaded" + +- PR from a fork: OIDC is unavailable; the upload step is skipped. +- Same-repo: confirm `AWS_ROLE_ARN` secret is set and the role's trust + policy allows `repo:ROCm/TransferBench:*`. + +### Build fails: missing `libibverbs.h` / `mpi.h` + +The packaged builds disable both `ENABLE_NIC_EXEC` and `ENABLE_MPI_COMM`, so these +headers are not required. If you've manually re-enabled either flag for a local +build, install the dev packages yourself: + +```bash +# Ubuntu — for ENABLE_NIC_EXEC=ON +sudo apt install -y libibverbs-dev rdma-core +# Ubuntu — for ENABLE_MPI_COMM=ON +sudo apt install -y libopenmpi-dev openmpi-bin +# Rocky/RHEL +sudo dnf install -y rdma-core-devel openmpi-devel +``` + +### TheRock tarball download 404s + +Check available builds at +. Set +`ROCM_VERSION` explicitly to a known-good version. + +## References + +- [TheRock Releases](https://github.com/ROCm/TheRock/blob/main/RELEASES.md) +- [TheRock nightly tarballs](https://therock-nightly-tarball.s3.amazonaws.com/index.html) +- [ROCmValidationSuite packaging workflow](https://github.com/ROCm/ROCmValidationSuite/blob/master/.github/workflows/README_BUILD_PACKAGES.md) — reference implementation +- [TransferBench README](../../README.md) diff --git a/.github/workflows/build-relocatable-packages.yml b/.github/workflows/build-relocatable-packages.yml new file mode 100644 index 00000000..af90b466 --- /dev/null +++ b/.github/workflows/build-relocatable-packages.yml @@ -0,0 +1,301 @@ +name: Build Relocatable Packages + +on: + push: + branches: [develop, mainline, 'release/**', candidate] + pull_request: + branches: [develop, mainline] + schedule: + # Daily at 13:00 UTC (5:00 AM PST) + - cron: '0 13 * * *' + workflow_dispatch: + inputs: + rocm_version: + description: 'ROCm version (empty = auto-fetch latest from TheRock)' + required: false + default: '' + gpu_family: + description: 'GPU family target' + required: false + default: 'gfx94X-dcgpu' + type: choice + options: + - gfx94X-dcgpu + - gfx950-dcgpu + - gfx110X-all + - gfx120X-all + - gfx1151 + +permissions: + contents: read + id-token: write # Required for OIDC S3 upload + +env: + ROCM_VERSION: ${{ github.event.inputs.rocm_version || '' }} + GPU_FAMILY: ${{ github.event.inputs.gpu_family || 'gfx94X-dcgpu' }} + BUILD_TYPE: Release + +jobs: + # ============================================================ + # Ubuntu 22.04 — DEB + TGZ + # ============================================================ + build-ubuntu: + name: Build (Ubuntu 22.04) + runs-on: ${{ vars.RUNNER_LABEL || 'ubuntu-22.04' }} + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + submodules: recursive + fetch-depth: 0 # for branch.commit version tags + + - name: Set environment + run: | + echo "ROCM_VERSION=${ROCM_VERSION}" >> "$GITHUB_ENV" + echo "GPU_FAMILY=${GPU_FAMILY}" >> "$GITHUB_ENV" + echo "BUILD_TYPE=${BUILD_TYPE}" >> "$GITHUB_ENV" + + - name: Build packages + run: | + chmod +x ./build_packages_local.sh + sudo -E ROCM_VERSION="${ROCM_VERSION}" \ + GPU_FAMILY="${GPU_FAMILY}" \ + BUILD_TYPE="${BUILD_TYPE}" \ + GITHUB_RUN_NUMBER="${GITHUB_RUN_NUMBER}" \ + GITHUB_REF_NAME="${GITHUB_REF_NAME}" \ + ./build_packages_local.sh + + - name: Verify DEB package + run: | + shopt -s nullglob + for deb in build/amdrocm*-transferbench*.deb; do + echo "==> ${deb}" + dpkg-deb -I "${deb}" + dpkg-deb -c "${deb}" | head -50 + done + + - name: Upload artifacts (always, for inspection) + uses: actions/upload-artifact@v4 + with: + name: ubuntu-22.04-packages + path: | + build/amdrocm*-transferbench*.deb + build/amdrocm*-transferbench*.tar.gz + if-no-files-found: error + + - name: Configure AWS credentials (OIDC) + if: github.repository == 'ROCm/TransferBench' && vars.AWS_S3_BUCKET != '' + uses: aws-actions/configure-aws-credentials@v4 + with: + role-to-assume: ${{ secrets.AWS_ROLE_ARN }} + aws-region: us-east-1 + + - name: Upload to S3 + if: github.repository == 'ROCm/TransferBench' && vars.AWS_S3_BUCKET != '' + env: + AWS_S3_BUCKET: ${{ vars.AWS_S3_BUCKET }} + run: | + set -euo pipefail + if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then + S3_PATH="s3://${AWS_S3_BUCKET}/transferbench/${GITHUB_HEAD_REF//\//_}/${GITHUB_RUN_NUMBER}/ubuntu-22.04" + METADATA="skip" + elif [[ "${GITHUB_REF_NAME}" == release/* ]]; then + S3_PATH="s3://${AWS_S3_BUCKET}/release/transferbench/deb" + METADATA="generate" + else + S3_PATH="s3://${AWS_S3_BUCKET}/nightly/transferbench/deb" + METADATA="generate" + fi + echo "S3_DEB_PATH=${S3_PATH}" >> "$GITHUB_ENV" + echo "DEB_METADATA=${METADATA}" >> "$GITHUB_ENV" + aws s3 cp build/ "${S3_PATH}/" --recursive --exclude "*" \ + --include "amdrocm*-transferbench*.deb" \ + --include "amdrocm*-transferbench*.tar.gz" + echo "Uploaded to ${S3_PATH}" + + - name: Generate apt repo metadata + if: github.repository == 'ROCm/TransferBench' && env.DEB_METADATA == 'generate' + env: + AWS_S3_BUCKET: ${{ vars.AWS_S3_BUCKET }} + run: | + set -euo pipefail + WORK="$(mktemp -d)" + echo "Downloading existing DEBs from ${S3_DEB_PATH}/ ..." + aws s3 sync "${S3_DEB_PATH}/" "${WORK}/" \ + --exclude "*" --include "*.deb" \ + --no-progress + cp build/amdrocm*-transferbench*.deb "${WORK}/" 2>/dev/null || true + cd "${WORK}" + dpkg-scanpackages --multiversion . /dev/null > Packages + gzip -k -f Packages + { + echo "Origin: ROCm-TransferBench" + echo "Label: ROCm TransferBench Packages" + echo "Suite: stable" + echo "Codename: stable" + echo "Architectures: amd64" + echo "Components: main" + echo "Description: TransferBench DEB packages built from TheRock SDK" + echo "Date: $(date -Ru)" + } > Release + aws s3 sync "${WORK}/" "${S3_DEB_PATH}/" --no-progress + echo "=== DEB repo metadata uploaded to ${S3_DEB_PATH}/ ===" + aws s3 ls "${S3_DEB_PATH}/" --human-readable + + # ============================================================ + # manylinux_2_28 (AlmaLinux 8) — RPM + TGZ + # ============================================================ + build-manylinux: + name: Build (manylinux_2_28) + runs-on: ${{ vars.RUNNER_LABEL_CONTAINER || 'ubuntu-latest' }} + container: + image: quay.io/pypa/manylinux_2_28_x86_64 + steps: + - name: Checkout + uses: actions/checkout@v4 + with: + submodules: recursive + fetch-depth: 0 + + - name: Set environment + run: | + echo "ROCM_VERSION=${ROCM_VERSION}" >> "$GITHUB_ENV" + echo "GPU_FAMILY=${GPU_FAMILY}" >> "$GITHUB_ENV" + echo "BUILD_TYPE=${BUILD_TYPE}" >> "$GITHUB_ENV" + + - name: Build packages + run: | + chmod +x ./build_packages_local.sh + # No sudo: container runs as root + ROCM_VERSION="${ROCM_VERSION}" \ + GPU_FAMILY="${GPU_FAMILY}" \ + BUILD_TYPE="${BUILD_TYPE}" \ + GITHUB_RUN_NUMBER="${GITHUB_RUN_NUMBER}" \ + GITHUB_REF_NAME="${GITHUB_REF_NAME}" \ + ./build_packages_local.sh + + - name: Verify RPM package + run: | + shopt -s nullglob + for rpm in build/amdrocm*-transferbench*.rpm; do + echo "==> ${rpm}" + rpm -qip "${rpm}" + rpm -qlp "${rpm}" | head -50 + done + + - name: Upload artifacts (always, for inspection) + uses: actions/upload-artifact@v4 + with: + name: manylinux_2_28-packages + path: | + build/amdrocm*-transferbench*.rpm + build/amdrocm*-transferbench*.tar.gz + if-no-files-found: error + + - name: Install AWS CLI + if: github.repository == 'ROCm/TransferBench' && vars.AWS_S3_BUCKET != '' + run: | + curl -fsSL "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o /tmp/awscli.zip + (cd /tmp && unzip -q awscli.zip && ./aws/install) + + - name: Configure AWS credentials (OIDC) + if: github.repository == 'ROCm/TransferBench' && vars.AWS_S3_BUCKET != '' + uses: aws-actions/configure-aws-credentials@v4 + with: + role-to-assume: ${{ secrets.AWS_ROLE_ARN }} + aws-region: us-east-1 + + - name: Upload to S3 + if: github.repository == 'ROCm/TransferBench' && vars.AWS_S3_BUCKET != '' + env: + AWS_S3_BUCKET: ${{ vars.AWS_S3_BUCKET }} + run: | + set -euo pipefail + if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then + S3_RPM="s3://${AWS_S3_BUCKET}/transferbench/${GITHUB_HEAD_REF//\//_}/${GITHUB_RUN_NUMBER}/manylinux_2_28" + S3_TAR="${S3_RPM}" + METADATA="skip" + elif [[ "${GITHUB_REF_NAME}" == release/* ]]; then + S3_RPM="s3://${AWS_S3_BUCKET}/release/transferbench/rpm" + S3_TAR="s3://${AWS_S3_BUCKET}/release/transferbench/tar" + METADATA="generate" + else + S3_RPM="s3://${AWS_S3_BUCKET}/nightly/transferbench/rpm" + S3_TAR="s3://${AWS_S3_BUCKET}/nightly/transferbench/tar" + METADATA="generate" + fi + echo "S3_RPM_PATH=${S3_RPM}" >> "$GITHUB_ENV" + echo "S3_TAR_PATH=${S3_TAR}" >> "$GITHUB_ENV" + echo "RPM_METADATA=${METADATA}" >> "$GITHUB_ENV" + aws s3 cp build/ "${S3_RPM}/" --recursive --exclude "*" --include "amdrocm*-transferbench*.rpm" + aws s3 cp build/ "${S3_TAR}/" --recursive --exclude "*" --include "amdrocm*-transferbench*.tar.gz" + echo "Uploaded RPM to ${S3_RPM}, TGZ to ${S3_TAR}" + + - name: Generate yum repo metadata + if: github.repository == 'ROCm/TransferBench' && env.RPM_METADATA == 'generate' + env: + AWS_S3_BUCKET: ${{ vars.AWS_S3_BUCKET }} + run: | + set -euo pipefail + dnf install -y createrepo_c || yum install -y createrepo_c + WORK="$(mktemp -d)" + echo "Downloading existing RPMs from s3://${AWS_S3_BUCKET}/${S3_RPM_PATH#s3://${AWS_S3_BUCKET}/}/ ..." + aws s3 sync "${S3_RPM_PATH}/" "${WORK}/" --exclude "repodata/*" --no-progress + cp build/amdrocm*-transferbench*.rpm "${WORK}/" 2>/dev/null || true + createrepo_c "${WORK}" + aws s3 sync "${WORK}/" "${S3_RPM_PATH}/" --no-progress + echo "=== RPM repository contents (RPMs + repodata) synced to ${S3_RPM_PATH}/ ===" + aws s3 ls "${S3_RPM_PATH}/repodata/" --human-readable + + # ============================================================ + # Build report — collects S3 paths for browsing + # ============================================================ + release-summary: + name: Build Report + needs: [build-ubuntu, build-manylinux] + if: always() + runs-on: ${{ vars.RUNNER_LABEL_UTILITY || 'ubuntu-latest' }} + steps: + - name: Generate report + env: + AWS_S3_BUCKET: ${{ vars.AWS_S3_BUCKET }} + run: | + set -euo pipefail + mkdir -p report + { + echo "# TransferBench Build Report" + echo "" + echo "- Event: \`${GITHUB_EVENT_NAME}\`" + echo "- Ref: \`${GITHUB_REF_NAME}\`" + echo "- Run number: \`${GITHUB_RUN_NUMBER}\`" + echo "- ROCm: \`${ROCM_VERSION:-auto}\`" + echo "- GPU family: \`${GPU_FAMILY}\`" + echo "- Ubuntu job: \`${{ needs.build-ubuntu.result }}\`" + echo "- manylinux: \`${{ needs.build-manylinux.result }}\`" + echo "" + if [[ -n "${AWS_S3_BUCKET:-}" ]]; then + echo "## S3 Upload Locations" + if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then + BASE="transferbench/${GITHUB_HEAD_REF//\//_}/${GITHUB_RUN_NUMBER}" + echo "- [DEB (Ubuntu)](https://s3.console.aws.amazon.com/s3/buckets/${AWS_S3_BUCKET}?prefix=${BASE}/ubuntu-22.04/)" + echo "- [RPM/TGZ (manylinux)](https://s3.console.aws.amazon.com/s3/buckets/${AWS_S3_BUCKET}?prefix=${BASE}/manylinux_2_28/)" + elif [[ "${GITHUB_REF_NAME}" == release/* ]]; then + echo "- [DEB](https://s3.console.aws.amazon.com/s3/buckets/${AWS_S3_BUCKET}?prefix=release/transferbench/deb/)" + echo "- [RPM](https://s3.console.aws.amazon.com/s3/buckets/${AWS_S3_BUCKET}?prefix=release/transferbench/rpm/)" + echo "- [TGZ](https://s3.console.aws.amazon.com/s3/buckets/${AWS_S3_BUCKET}?prefix=release/transferbench/tar/)" + else + echo "- [DEB](https://s3.console.aws.amazon.com/s3/buckets/${AWS_S3_BUCKET}?prefix=nightly/transferbench/deb/)" + echo "- [RPM](https://s3.console.aws.amazon.com/s3/buckets/${AWS_S3_BUCKET}?prefix=nightly/transferbench/rpm/)" + echo "- [TGZ](https://s3.console.aws.amazon.com/s3/buckets/${AWS_S3_BUCKET}?prefix=nightly/transferbench/tar/)" + fi + else + echo "_S3 upload not configured (\`AWS_S3_BUCKET\` variable not set)._" + fi + } > report/build-report.md + cat report/build-report.md >> "$GITHUB_STEP_SUMMARY" + + - name: Upload report + uses: actions/upload-artifact@v4 + with: + name: build-report + path: report/build-report.md diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 00000000..6d3ba2f9 --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,71 @@ +name: "CodeQL Security Scanning" + +on: + push: + branches: + - develop + - mainline + - candidate + pull_request: + branches: + - develop + - mainline + schedule: + # Weekly scan on Fridays at 6:34 PM UTC + - cron: '34 18 * * 5' + +jobs: + analyze: + name: Analyze (C/C++) + runs-on: ubuntu-latest + container: rocm/dev-ubuntu-22.04:latest + permissions: + security-events: write + packages: read + actions: read + contents: read + + steps: + - name: Install OS requirements + timeout-minutes: 10 + run: | + apt update + apt install -y software-properties-common + apt-add-repository -y ppa:git-core/ppa + apt-get update + apt install -y git + + - name: Checkout repository + uses: actions/checkout@v4 + + - name: Configure Git Safe Directory + run: | + git config --global --add safe.directory "${GITHUB_WORKSPACE}" + + - name: Install dependencies + run: | + apt-get update + apt-get install -y build-essential cmake libnuma-dev + + - name: Initialize CodeQL + uses: github/codeql-action/init@v3 + with: + languages: c-cpp + build-mode: manual + queries: security-extended + + - name: Build TransferBench + run: | + mkdir -p build + cd build + cmake .. -DCMAKE_BUILD_TYPE=Release \ + -DBUILD_LOCAL_GPU_TARGET_ONLY=ON \ + -DENABLE_NIC_EXEC=OFF \ + -DENABLE_MPI_COMM=OFF \ + -DCMAKE_PREFIX_PATH=/opt/rocm + make -j $(nproc) + + - name: Perform CodeQL Analysis + uses: github/codeql-action/analyze@v3 + with: + category: "/language:c-cpp" diff --git a/CMakeLists.txt b/CMakeLists.txt index 92ce4a38..0646b67d 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -102,7 +102,42 @@ set(ENV{ROCM_PATH} "${ROCM_PATH}") # TransferBench project definitions #================================================================================================== -set(VERSION_STRING "1.67.00") +set(TRANSFERBENCH_VERSION_MAJOR 1) +set(TRANSFERBENCH_VERSION_MINOR 67) +set(TRANSFERBENCH_VERSION_PATCH_FALLBACK "00") + +# Auto-compute patch from git: count commits since the last v..* tag. +# Falls back to TRANSFERBENCH_VERSION_PATCH_FALLBACK when git is unavailable, +# this is not a git checkout, or no matching tag exists. Mirrors the RVS flow. +set(TRANSFERBENCH_VERSION_PATCH "${TRANSFERBENCH_VERSION_PATCH_FALLBACK}") +find_package(Git QUIET) +if(GIT_FOUND AND EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/.git") + execute_process( + COMMAND "${GIT_EXECUTABLE}" describe --tags --abbrev=0 --match + "v${TRANSFERBENCH_VERSION_MAJOR}.${TRANSFERBENCH_VERSION_MINOR}.*" + WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" + OUTPUT_VARIABLE _tb_last_tag + OUTPUT_STRIP_TRAILING_WHITESPACE + RESULT_VARIABLE _tb_describe_rc + ERROR_QUIET) + if(_tb_describe_rc EQUAL 0 AND _tb_last_tag) + execute_process( + COMMAND "${GIT_EXECUTABLE}" rev-list --count "${_tb_last_tag}..HEAD" + WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" + OUTPUT_VARIABLE _tb_commit_count + OUTPUT_STRIP_TRAILING_WHITESPACE + RESULT_VARIABLE _tb_count_rc + ERROR_QUIET) + if(_tb_count_rc EQUAL 0 AND _tb_commit_count MATCHES "^[0-9]+$") + set(TRANSFERBENCH_VERSION_PATCH "${_tb_commit_count}") + endif() + endif() +endif() + +set(VERSION_STRING + "${TRANSFERBENCH_VERSION_MAJOR}.${TRANSFERBENCH_VERSION_MINOR}.${TRANSFERBENCH_VERSION_PATCH}") +message(STATUS "TransferBench version: ${VERSION_STRING}") + project(TransferBench VERSION ${VERSION_STRING} LANGUAGES CXX) if(NOT CMAKE_BUILD_TYPE AND NOT CMAKE_CONFIGURATION_TYPES) @@ -128,6 +163,7 @@ option(ENABLE_MPI_COMM "Enable MPI Communicator support" option(ENABLE_DMA_BUF "Enable DMA-BUF support for GPU Direct RDMA" OFF) option(ENABLE_AMD_SMI "Enable AMD-SMI pod membership queries" OFF) option(ENABLE_POD_COMM "Enable pod communication" OFF) +option(BUILD_RELOCATABLE_PACKAGE "Build with RVS-style relocatable RPATH and amdrocm-transferbench package naming" OFF) include(cmake/Dependencies.cmake) # rocm-cmake, rocm_local_targets, rocm_check_target_ids @@ -475,16 +511,148 @@ target_link_libraries(TransferBench PRIVATE hip::host hip::device dl) target_link_libraries(TransferBench PRIVATE hsa-runtime64) target_link_libraries(TransferBench PRIVATE numa) -rocm_install(TARGETS TransferBench COMPONENT devel) -rocm_setup_version(VERSION ${VERSION_STRING}) +# gcc <9 ships std::filesystem in a separate library (libstdc++fs). +# Required on AlmaLinux 8 / manylinux_2_28; harmless no-op stub on newer toolchains. +target_link_libraries(TransferBench PRIVATE stdc++fs) + +if(BUILD_RELOCATABLE_PACKAGE) + # RVS-style relocatable packaging: bypass rocm_install/rocm_create_package and + # drive CPack directly so CMAKE_INSTALL_PREFIX / CPACK_PACKAGING_INSTALL_PREFIX + # set by the caller (build_packages_local.sh) are honored. + # + # The CPack flow below uses CPACK_ARCHIVE_FILE_NAME (3.13+) and the + # DEB-DEFAULT / RPM-DEFAULT canonical-naming sentinels (3.6+). The + # project-wide cmake_minimum_required is 3.5 for the non-packaging build, so + # gate the relocatable path locally to avoid silently producing a literal + # "DEB-DEFAULT" filename on a too-old CMake. + if(CMAKE_VERSION VERSION_LESS 3.13) + message(FATAL_ERROR + "BUILD_RELOCATABLE_PACKAGE requires CMake >= 3.13 " + "(found ${CMAKE_VERSION}); needed for CPACK_ARCHIVE_FILE_NAME and " + "the DEB-DEFAULT / RPM-DEFAULT canonical-naming sentinels.") + endif() + if(NOT DEFINED ROCM_MAJOR_VERSION) + set(ROCM_MAJOR_VERSION "7") + endif() -rocm_package_add_dependencies(DEPENDS "numactl" "hsa-rocr") + # Relocatable RPATH (matches the RVS reference flow). Mirrors what + # build_packages_local.sh used to inject via -DCMAKE_INSTALL_RPATH=, so + # plain `cmake -DBUILD_RELOCATABLE_PACKAGE=ON ..` now produces the same + # RPATH as a CI/packaged build. + set(CMAKE_SKIP_RPATH FALSE) + set(CMAKE_INSTALL_RPATH_USE_LINK_PATH FALSE) + set(CMAKE_INSTALL_RPATH + "\$ORIGIN:\$ORIGIN/../lib:/opt/rocm/extras-${ROCM_MAJOR_VERSION}/lib:/opt/rocm/lib:/opt/rocm/lib/llvm/lib:/opt/rocm/core-${ROCM_MAJOR_VERSION}/lib:/opt/rocm/core-${ROCM_MAJOR_VERSION}/lib/llvm/lib") + set(CMAKE_BUILD_RPATH "${CMAKE_INSTALL_RPATH}") + # Strip implicit SDK-from-build-host paths on install so the ephemeral + # $HOME/rocm-sdk/install path the CI script uses does not leak into the + # packaged binary's RPATH. + if(NOT CMAKE_VERSION VERSION_LESS "3.16") + set(CMAKE_INSTALL_REMOVE_ENVIRONMENT_RPATH TRUE) + endif() -set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_CURRENT_SOURCE_DIR}/LICENSE.md") -set(CPACK_RPM_PACKAGE_LICENSE "MIT") + install(TARGETS TransferBench RUNTIME DESTINATION bin COMPONENT devel) + + set(CPACK_PACKAGE_NAME "amdrocm${ROCM_MAJOR_VERSION}-transferbench") + set(CPACK_PACKAGE_VERSION "${VERSION_STRING}") + set(CPACK_PACKAGE_VENDOR "Advanced Micro Devices, Inc.") + set(CPACK_PACKAGE_CONTACT "RCCL Team ") + set(CPACK_PACKAGE_DESCRIPTION_SUMMARY "TransferBench: benchmark simultaneous transfers between CPU/GPU/NIC") + set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_CURRENT_SOURCE_DIR}/LICENSE.md") + + # Per-build release tag, threaded into DEB/RPM metadata AND the TGZ filename. + # Prefer the explicit -D from build_packages_local.sh; fall back to the env + # vars CPack itself reads (so direct cmake invocations still work). + # Quote variable references throughout so the comparison is unambiguous + # under any CMP0054 setting. + set(_tb_pkg_release "") + if(NOT "${TRANSFERBENCH_PACKAGE_RELEASE}" STREQUAL "") + set(_tb_pkg_release "${TRANSFERBENCH_PACKAGE_RELEASE}") + elseif(NOT "$ENV{CPACK_RPM_PACKAGE_RELEASE}" STREQUAL "") + set(_tb_pkg_release "$ENV{CPACK_RPM_PACKAGE_RELEASE}") + elseif(NOT "$ENV{CPACK_DEBIAN_PACKAGE_RELEASE}" STREQUAL "") + set(_tb_pkg_release "$ENV{CPACK_DEBIAN_PACKAGE_RELEASE}") + endif() -rocm_create_package( - NAME ${LIBRARY_NAME} - DESCRIPTION "TransferBench package" - MAINTAINER "RCCL Team " -) + # DEB + set(CPACK_DEBIAN_PACKAGE_NAME "${CPACK_PACKAGE_NAME}") + set(CPACK_DEBIAN_PACKAGE_ARCHITECTURE "amd64") + set(CPACK_DEBIAN_PACKAGE_DEPENDS "numactl, libnuma1") + set(CPACK_DEBIAN_PACKAGE_MAINTAINER "${CPACK_PACKAGE_CONTACT}") + if(NOT "${_tb_pkg_release}" STREQUAL "") + set(CPACK_DEBIAN_PACKAGE_RELEASE "${_tb_pkg_release}") + elseif(DEFINED ENV{CPACK_DEBIAN_PACKAGE_RELEASE}) + set(CPACK_DEBIAN_PACKAGE_RELEASE "$ENV{CPACK_DEBIAN_PACKAGE_RELEASE}") + endif() + # Canonical filename: _-_.deb + set(CPACK_DEBIAN_FILE_NAME "DEB-DEFAULT") + + # RPM + set(CPACK_RPM_PACKAGE_NAME "${CPACK_PACKAGE_NAME}") + set(CPACK_RPM_PACKAGE_LICENSE "MIT") + set(CPACK_RPM_PACKAGE_REQUIRES "numactl") + set(CPACK_RPM_PACKAGE_VENDOR "${CPACK_PACKAGE_VENDOR}") + if(NOT "${_tb_pkg_release}" STREQUAL "") + set(CPACK_RPM_PACKAGE_RELEASE "${_tb_pkg_release}") + elseif(DEFINED ENV{CPACK_RPM_PACKAGE_RELEASE}) + set(CPACK_RPM_PACKAGE_RELEASE "$ENV{CPACK_RPM_PACKAGE_RELEASE}") + endif() + # Canonical filename: --..rpm + set(CPACK_RPM_FILE_NAME "RPM-DEFAULT") + # Use the actual install prefix (caller-controlled in relocatable mode) + # rather than hard-coded /opt/... paths. + if(DEFINED CPACK_PACKAGING_INSTALL_PREFIX) + set(_rpm_exclude_prefix "${CPACK_PACKAGING_INSTALL_PREFIX}") + else() + set(_rpm_exclude_prefix "${CMAKE_INSTALL_PREFIX}") + endif() + set(CPACK_RPM_EXCLUDE_FROM_AUTO_FILELIST_ADDITION + "/opt" "/opt/rocm" + "${_rpm_exclude_prefix}" + "${_rpm_exclude_prefix}/bin") + + # Advisory install-time check for libhsa-runtime64.so.1. The package declares + # no hard ROCm dep so it can install on TheRock-tarball systems where no ROCm + # component is tracked by apt/dpkg; the postinst warns (never fails) when the + # HSA runtime is not discoverable, so a missing runtime surfaces at install + # time instead of as a dynamic-linker error on first invocation. + set(_tb_postinst_src "${CMAKE_CURRENT_SOURCE_DIR}/packaging/postinst-check-hsa.sh") + set(_tb_postinst_deb "${CMAKE_BINARY_DIR}/packaging/postinst") + configure_file("${_tb_postinst_src}" "${_tb_postinst_deb}" COPYONLY) + set(CPACK_DEBIAN_PACKAGE_CONTROL_EXTRA "${_tb_postinst_deb}") + set(CPACK_RPM_POST_INSTALL_SCRIPT_FILE "${_tb_postinst_src}") + + # TGZ — embed release tag so successive runs do not collide on the same key. + # CMake 3.13+ honors CPACK_ARCHIVE_FILE_NAME for archive generators, but + # CMake 3.22 (Ubuntu 22.04) falls back to CPACK_PACKAGE_FILE_NAME for TGZ. + # Set both to the same suffixed value. DEB/RPM are unaffected because they + # use the explicit DEB-DEFAULT / RPM-DEFAULT canonical-naming tokens above. + if("${_tb_pkg_release}" STREQUAL "") + set(_tb_archive_name "${CPACK_PACKAGE_NAME}-${CPACK_PACKAGE_VERSION}-Linux") + else() + set(_tb_archive_name "${CPACK_PACKAGE_NAME}-${CPACK_PACKAGE_VERSION}-${_tb_pkg_release}-Linux") + endif() + set(CPACK_ARCHIVE_FILE_NAME "${_tb_archive_name}") + set(CPACK_PACKAGE_FILE_NAME "${_tb_archive_name}") + + set(CPACK_GENERATOR "DEB;RPM;TGZ") + include(CPack) +else() + rocm_install(TARGETS TransferBench COMPONENT devel) + rocm_setup_version(VERSION ${VERSION_STRING}) + + # Package specific CPACK vars + rocm_package_add_dependencies(DEPENDS "numactl") + + set(CPACK_RESOURCE_FILE_LICENSE "${CMAKE_CURRENT_SOURCE_DIR}/LICENSE.md") + set(CPACK_RPM_PACKAGE_LICENSE "MIT") + + set(PACKAGE_NAME TB) + set(LIBRARY_NAME TransferBench) + + rocm_create_package( + NAME ${LIBRARY_NAME} + DESCRIPTION "TransferBench package" + MAINTAINER "RCCL Team " + ) +endif() diff --git a/build_packages_local.sh b/build_packages_local.sh new file mode 100755 index 00000000..91ace56b --- /dev/null +++ b/build_packages_local.sh @@ -0,0 +1,263 @@ +#!/usr/bin/env bash +# +# Copyright (c) Advanced Micro Devices, Inc. All rights reserved. +# +# build_packages_local.sh — single source of truth for building relocatable +# TransferBench packages (DEB / RPM / TGZ) against TheRock ROCm SDK. +# Used by both local developers and the GitHub Actions workflow. +# +# Usage: +# sudo ./build_packages_local.sh +# sudo -E ROCM_VERSION=7.11.0a20260121 GPU_FAMILY=gfx94X-dcgpu ./build_packages_local.sh +# +# Requires root (installs system packages). + +set -euo pipefail + +# -------- pretty output -------- +RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m'; BLUE='\033[0;34m'; NC='\033[0m' +log() { echo -e "${BLUE}[INFO]${NC} $*"; } +ok() { echo -e "${GREEN}[ OK ]${NC} $*"; } +warn() { echo -e "${YELLOW}[WARN]${NC} $*"; } +err() { echo -e "${RED}[FAIL]${NC} $*" >&2; } + +trap 'err "Build failed at line $LINENO"' ERR + +# -------- root check -------- +if [[ ${EUID} -ne 0 ]]; then + err "This script installs system packages and must run as root. Re-run with: sudo -E $0" + exit 1 +fi + +# -------- inputs -------- +ROCM_VERSION="${ROCM_VERSION:-}" # empty => auto-fetch latest +GPU_FAMILY="${GPU_FAMILY:-gfx94X-dcgpu}" +BUILD_TYPE="${BUILD_TYPE:-Release}" +GITHUB_RUN_NUMBER="${GITHUB_RUN_NUMBER:-1}" + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +BUILD_DIR="${REPO_ROOT}/build" +SDK_DIR="${HOME}/rocm-sdk" +ROCM_PATH="${SDK_DIR}/install" + +# Containerized builds (e.g. manylinux on a host-mounted workspace) hit git's +# "dubious ownership" guard because the checkout is host-UID-owned but we run +# as root. Without this, `git describe` in CMakeLists.txt silently fails and +# TRANSFERBENCH_VERSION_PATCH falls back to its hard-coded default. +# +# Use GIT_CONFIG_* env vars (git >= 2.31) so the scoped safe.directory entry +# is inherited by CMake's `execute_process(git …)` children without touching +# the user's persistent ~/.gitconfig (especially harmful under sudo, where +# the modification would land in root's global config). +export GIT_CONFIG_COUNT=1 +export GIT_CONFIG_KEY_0="safe.directory" +export GIT_CONFIG_VALUE_0="${REPO_ROOT}" + +# Default GPU targets baked into every package, regardless of GPU_FAMILY tarball. +DEFAULT_GPU_TARGETS="gfx906;gfx908;gfx90a;gfx942;gfx950;gfx1030;gfx1100;gfx1101;gfx1102;gfx1150;gfx1151;gfx1200;gfx1201" +GPU_TARGETS="${GPU_TARGETS:-$DEFAULT_GPU_TARGETS}" + +# -------- detect OS -------- +if [[ -f /etc/os-release ]]; then + # shellcheck disable=SC1091 + . /etc/os-release + OS_ID="${ID:-unknown}" + OS_LIKE="${ID_LIKE:-}" +else + err "/etc/os-release not found; cannot detect distro"; exit 1 +fi + +case "${OS_ID}:${OS_LIKE}" in + ubuntu:*|debian:*|*:*debian*) DISTRO="ubuntu" ;; + almalinux:*|rocky:*|rhel:*|centos:*|*:*rhel*|*:*fedora*) DISTRO="almalinux" ;; + *) + if command -v apt-get >/dev/null 2>&1; then DISTRO="ubuntu" + elif command -v yum >/dev/null 2>&1 || command -v dnf >/dev/null 2>&1; then DISTRO="almalinux" + else err "Unsupported distro: ${OS_ID}"; exit 1 + fi + ;; +esac +log "Detected distro: ${DISTRO} (${OS_ID})" + +# -------- install dependencies -------- +log "Installing build dependencies..." +if [[ "${DISTRO}" == "ubuntu" ]]; then + export DEBIAN_FRONTEND=noninteractive + apt-get update -y + apt-get install -y --no-install-recommends \ + build-essential cmake git curl tar xz-utils ca-certificates pkg-config \ + python3 python3-pip \ + libnuma-dev \ + dpkg-dev rpm file apt-utils + CMAKE_BIN="cmake" + CMAKE_CXX_COMPILER_OVERRIDE="" +else + # AlmaLinux / Rocky / RHEL / manylinux_2_28 + if command -v dnf >/dev/null 2>&1; then PKG="dnf"; else PKG="yum"; fi + ${PKG} install -y epel-release || true + # Enable PowerTools/CRB for createrepo_c, etc. + ${PKG} config-manager --set-enabled powertools 2>/dev/null \ + || ${PKG} config-manager --set-enabled crb 2>/dev/null || true + ${PKG} install -y \ + gcc gcc-c++ make cmake3 git curl tar xz ca-certificates pkgconfig \ + python3 python3-pip \ + numactl-devel \ + rpm-build dpkg createrepo_c file + CMAKE_BIN="cmake3" + command -v cmake3 >/dev/null 2>&1 || CMAKE_BIN="cmake" + CMAKE_CXX_COMPILER_OVERRIDE="${ROCM_PATH}/bin/hipcc" +fi +ok "Dependencies installed" + +# -------- fetch ROCm SDK from TheRock -------- +TARBALL_BASE="https://therock-nightly-tarball.s3.amazonaws.com" +TAR_PREFIX="therock-dist-linux-${GPU_FAMILY}-" + +if [[ -z "${ROCM_VERSION}" ]]; then + log "ROCM_VERSION not set; auto-fetching latest for ${GPU_FAMILY}..." + # No LATEST.txt is published; list the bucket and pick the highest version key. + LIST_URL="${TARBALL_BASE}/?list-type=2&max-keys=1000&prefix=${TAR_PREFIX}" + # Filter to versioned tarballs only (skip ADHOCBUILD-* and other non-release keys); + # match: ..<...>.tar.gz + LATEST_KEY="$(curl -fsSL "${LIST_URL}" 2>/dev/null \ + | tr '<' '\n' \ + | sed -n 's|^Key>||p' \ + | grep -E "^${TAR_PREFIX}[0-9]+\.[0-9]+\.[0-9a-z]+\.tar\.gz$" \ + | sort -V \ + | tail -1 || true)" + if [[ -n "${LATEST_KEY}" ]]; then + ROCM_VERSION="${LATEST_KEY#${TAR_PREFIX}}" + ROCM_VERSION="${ROCM_VERSION%.tar.gz}" + ok "Latest ROCm version for ${GPU_FAMILY}: ${ROCM_VERSION}" + else + warn "Could not list ${LIST_URL}; falling back to pinned default" + ROCM_VERSION="7.13.0a20260423" + fi +fi + +TARBALL_NAME="${TAR_PREFIX}${ROCM_VERSION}.tar.gz" +TARBALL_URL="${TARBALL_BASE}/${TARBALL_NAME}" + +mkdir -p "${SDK_DIR}" +if [[ ! -d "${ROCM_PATH}" ]] || [[ ! -f "${SDK_DIR}/.installed-${ROCM_VERSION}-${GPU_FAMILY}" ]]; then + log "Downloading ${TARBALL_URL}..." + curl -fSL "${TARBALL_URL}" -o "${SDK_DIR}/${TARBALL_NAME}" + log "Extracting to ${SDK_DIR}..." + rm -rf "${ROCM_PATH}" + mkdir -p "${ROCM_PATH}" + tar -xzf "${SDK_DIR}/${TARBALL_NAME}" -C "${ROCM_PATH}" --strip-components=1 \ + || tar -xzf "${SDK_DIR}/${TARBALL_NAME}" -C "${ROCM_PATH}" + rm -f "${SDK_DIR}/${TARBALL_NAME}" + touch "${SDK_DIR}/.installed-${ROCM_VERSION}-${GPU_FAMILY}" + ok "ROCm SDK installed at ${ROCM_PATH}" +else + log "Reusing cached ROCm SDK at ${ROCM_PATH}" +fi + +export ROCM_PATH +export PATH="${ROCM_PATH}/bin:${PATH}" +export LD_LIBRARY_PATH="${ROCM_PATH}/lib:${LD_LIBRARY_PATH:-}" +export CMAKE_PREFIX_PATH="${ROCM_PATH}:${CMAKE_PREFIX_PATH:-}" + +# Locate HIP device libraries (amdgcn bitcode) +for candidate in \ + "${ROCM_PATH}/amdgcn/bitcode" \ + "${ROCM_PATH}/lib/llvm/amdgcn/bitcode" \ + "${ROCM_PATH}/lib/clang/amdgcn/bitcode"; do + if [[ -d "${candidate}" ]]; then export HIP_DEVICE_LIB_PATH="${candidate}"; break; fi +done +if [[ -n "${HIP_DEVICE_LIB_PATH:-}" ]]; then + ok "HIP_DEVICE_LIB_PATH=${HIP_DEVICE_LIB_PATH}" +else + warn "amdgcn bitcode directory not found under ${ROCM_PATH}; build may fail" +fi + +# -------- compute version helpers -------- +# ROCM_MAJOR / MINOR / patch helpers (e.g. 7.11.0a20260121 -> major=7 minor=11) +ROCM_MAJOR="$(echo "${ROCM_VERSION}" | sed -E 's/^([0-9]+)\..*/\1/')" +ROCM_MINOR="$(echo "${ROCM_VERSION}" | sed -E 's/^[0-9]+\.([0-9]+).*/\1/')" +printf -v ROCM_LIBPATCH_VERSION '%02d%02d' "${ROCM_MAJOR}" "${ROCM_MINOR}" +export ROCM_MAJOR ROCM_MINOR ROCM_LIBPATCH_VERSION +log "ROCm major=${ROCM_MAJOR} minor=${ROCM_MINOR} libpatch=${ROCM_LIBPATCH_VERSION}" + +# Package release string. Format mirrors the RVS reference flow: +# default (push/schedule/dispatch/local): r. +# pull request: r... +# release/* branch (non-PR): ${GITHUB_RUN_NUMBER} (fallback 1) +GIT_BRANCH="${GITHUB_REF_NAME:-$(git -C "${REPO_ROOT}" rev-parse --abbrev-ref HEAD 2>/dev/null || echo unknown)}" +GIT_COMMIT="$(git -C "${REPO_ROOT}" rev-parse --short HEAD 2>/dev/null || echo unknown)" +BUILD_DATE_UTC="$(date -u +%Y%m%d)" + +# Collapse non-alphanumerics into single dots and trim — DEB/RPM release +# fields reject most punctuation. +sanitize_release() { + local s + s="$(printf '%s' "$1" | sed -E 's/[^[:alnum:]]+/./g; s/^\.+//; s/\.+$//')" + printf '%s' "${s:-unknown}" +} + +if [[ "${GITHUB_EVENT_NAME:-}" == "pull_request" ]]; then + PR_BRANCH="$(sanitize_release "${GITHUB_HEAD_REF:-${GIT_BRANCH}}")" + PKG_RELEASE="r${ROCM_LIBPATCH_VERSION}.${BUILD_DATE_UTC}.${PR_BRANCH}.${GIT_COMMIT}" +elif [[ "${GIT_BRANCH}" == release/* ]]; then + PKG_RELEASE="${GITHUB_RUN_NUMBER:-1}" +else + PKG_RELEASE="r${ROCM_LIBPATCH_VERSION}.${BUILD_DATE_UTC}" +fi +export CPACK_DEBIAN_PACKAGE_RELEASE="${CPACK_DEBIAN_PACKAGE_RELEASE:-$PKG_RELEASE}" +export CPACK_RPM_PACKAGE_RELEASE="${CPACK_RPM_PACKAGE_RELEASE:-$PKG_RELEASE}" +log "Package release tag: ${PKG_RELEASE}" + +# -------- configure -------- +INSTALL_PREFIX="/opt/rocm/extras-${ROCM_MAJOR}" +# Relocatable RPATH defaults live in CMakeLists.txt under +# if(BUILD_RELOCATABLE_PACKAGE); enabling that option below activates them. + +log "Configuring CMake..." +rm -rf "${BUILD_DIR}" +mkdir -p "${BUILD_DIR}" + +CMAKE_ARGS=( + -B "${BUILD_DIR}" + -S "${REPO_ROOT}" + -DCMAKE_BUILD_TYPE="${BUILD_TYPE}" + -DROCM_PATH="${ROCM_PATH}" + -DROCM_MAJOR_VERSION="${ROCM_MAJOR}" + -DHIP_PLATFORM=amd + -DCMAKE_INSTALL_PREFIX="${INSTALL_PREFIX}" + -DCPACK_PACKAGING_INSTALL_PREFIX="${INSTALL_PREFIX}" + -DCMAKE_VERBOSE_MAKEFILE=ON + -DBUILD_RELOCATABLE_PACKAGE=ON + -DBUILD_LOCAL_GPU_TARGET_ONLY=OFF + -DENABLE_NIC_EXEC=OFF + -DENABLE_MPI_COMM=OFF + -DDISABLE_DMABUF=OFF + -DGPU_TARGETS="${GPU_TARGETS}" + -DTRANSFERBENCH_PACKAGE_RELEASE="${PKG_RELEASE}" +) +if [[ -n "${CMAKE_CXX_COMPILER_OVERRIDE}" ]]; then + CMAKE_ARGS+=(-DCMAKE_CXX_COMPILER="${CMAKE_CXX_COMPILER_OVERRIDE}") +fi + +"${CMAKE_BIN}" "${CMAKE_ARGS[@]}" +ok "CMake configured" + +# -------- build -------- +log "Building TransferBench (-j$(nproc))..." +"${CMAKE_BIN}" --build "${BUILD_DIR}" -- -j"$(nproc)" +ok "Build complete" + +# -------- package -------- +log "Packaging (DEB / RPM / TGZ via CPack)..." +pushd "${BUILD_DIR}" >/dev/null +if [[ "${DISTRO}" == "ubuntu" ]]; then + cpack -G DEB + cpack -G TGZ +else + cpack -G RPM + cpack -G TGZ +fi +popd >/dev/null + +ok "Packages written under ${BUILD_DIR}:" +ls -lh "${BUILD_DIR}"/amdrocm*-transferbench* 2>/dev/null || ls -lh "${BUILD_DIR}"/*.deb "${BUILD_DIR}"/*.rpm "${BUILD_DIR}"/*.tar.gz 2>/dev/null || true diff --git a/docs/install/INSTALL_TGZ.rst b/docs/install/INSTALL_TGZ.rst new file mode 100644 index 00000000..b365cef4 --- /dev/null +++ b/docs/install/INSTALL_TGZ.rst @@ -0,0 +1,136 @@ +:orphan: + +.. meta:: + :description: Install the relocatable TransferBench TGZ archive on any Linux distribution + :keywords: TransferBench, TGZ, tarball, install, relocatable + +.. _install-transferbench-tgz: + +------------------------------------------------ +Installing TransferBench from the TGZ archive +------------------------------------------------ + +The TransferBench TGZ archive (``amdrocm-transferbench-*.tar.gz``) is a +relocatable install tree that works on any Linux distribution where a +compatible ROCm runtime is already present. Use it when you cannot or do not +want to install the DEB or RPM package — for example on a distribution +without a native ROCm package, or inside a non-root container. + +The TGZ ships only the ``TransferBench`` binary and its supporting files. It +does **not** bundle ROCm; the host system must already provide the ROCm +runtime libraries (``hsa-rocr`` and the HIP runtime). + +Pre-install: ROCm +----------------- + +Install ROCm on the target system before extracting the TGZ. Follow the +official AMD documentation: + +* `ROCm documentation `_ +* `Linux install guide `_ + +After installing, ``ROCM_PATH`` (typically ``/opt/rocm``) must be set +correctly and the ROCm libraries must be loadable by the dynamic linker. + +Install runtime dependencies +---------------------------- + +The DEB and RPM packages declare these runtime dependencies; TGZ users must +install them manually on the target host. + +.. list-table:: + :header-rows: 1 + :widths: 30 70 + + * - Family + - Required packages + * - Debian / Ubuntu + - ``numactl``, ``libnuma1``, plus the ROCm runtime (``hsa-rocr``) + * - RHEL / Rocky / AlmaLinux + - ``numactl``, plus the ROCm runtime (``hsa-rocr``) + +Install commands: + +.. code-block:: bash + + # Ubuntu / Debian + sudo apt update && sudo apt install -y numactl libnuma1 + + # RHEL / Rocky / AlmaLinux + sudo dnf install -y numactl + +The ROCm packages (``hsa-rocr`` and friends) come from the ROCm repo +configured in the pre-install step above. + +Extract the TGZ +--------------- + +Extract the archive into ``/opt/rocm/extras-``, where ```` is +the ROCm major version the package was built against (encoded in the package +name, for example ``amdrocm7-transferbench-*.tar.gz`` → major ``7``). + +.. code-block:: bash + + # Example for ROCm major 7 — match your package + sudo mkdir -p /opt/rocm/extras-7 + sudo tar -xzf amdrocm7-transferbench-*.tar.gz -C /opt/rocm/extras-7 --strip-components=1 + +The ``--strip-components=1`` option discards the top-level directory inside +the tarball so files land directly under ``/opt/rocm/extras-7/{bin,lib,...}``. + +Configure ``PATH`` and ``LD_LIBRARY_PATH`` +------------------------------------------ + +Point the shell at the extracted prefix and your ROCm install. Copy and paste +the block as one unit (replace paths with your real ``ROCM_PATH`` and major +version): + +.. code-block:: bash + + export ROCM_PATH=/opt/rocm # or your real ROCm root + export PATH=/opt/rocm/extras-7/bin:$ROCM_PATH/bin:$PATH + export LD_LIBRARY_PATH=/opt/rocm/extras-7/lib:$ROCM_PATH/lib:$ROCM_PATH/lib/llvm/lib:$LD_LIBRARY_PATH + +The ``TransferBench`` binary embeds an ``RPATH`` covering ``$ORIGIN``, +``$ORIGIN/../lib``, ``/opt/rocm/extras-/lib``, ``/opt/rocm/lib``, +``/opt/rocm/lib/llvm/lib``, ``/opt/rocm/core-/lib``, and +``/opt/rocm/core-/lib/llvm/lib``. The ``LD_LIBRARY_PATH`` export above +is mainly defensive — useful if your ROCm tree lives somewhere non-standard +or if you want to override which copy of a library is loaded for +troubleshooting. + +Verify the install +------------------ + +.. code-block:: bash + + TransferBench + +Run with no arguments, ``TransferBench`` prints its version, usage, the +list of available preset benchmarks, and the detected GPU/CPU topology, +then exits. Seeing that output confirms the binary loaded its ROCm +libraries correctly. + +If the binary fails to load a shared library, inspect: + +.. code-block:: bash + + ldd /opt/rocm/extras-7/bin/TransferBench + readelf -d /opt/rocm/extras-7/bin/TransferBench | grep -E 'RPATH|RUNPATH' + +Make a persistent shell setup +----------------------------- + +To avoid re-exporting every shell, drop the variables into a profile script: + +.. code-block:: bash + + sudo tee /etc/profile.d/transferbench.sh >/dev/null <<'EOF' + export ROCM_PATH=/opt/rocm + export PATH=/opt/rocm/extras-7/bin:$ROCM_PATH/bin:$PATH + export LD_LIBRARY_PATH=/opt/rocm/extras-7/lib:$ROCM_PATH/lib:$ROCM_PATH/lib/llvm/lib:$LD_LIBRARY_PATH + EOF + sudo chmod 0644 /etc/profile.d/transferbench.sh + +Log out and back in (or ``source /etc/profile.d/transferbench.sh``) for the +changes to apply. diff --git a/docs/install/install.rst b/docs/install/install.rst index 4a44ff59..367fc307 100644 --- a/docs/install/install.rst +++ b/docs/install/install.rst @@ -83,3 +83,9 @@ TransferBench looks for NVCC in ``/usr/local/cuda`` by default. To modify the lo .. code-block:: bash CUDA_PATH=/usr/local/cuda make + +Installing from the relocatable TGZ archive +------------------------------------------- + +If you want to install a pre-built TransferBench binary on a system where +you cannot install the DEB or RPM package, see :ref:`install-transferbench-tgz`. diff --git a/packaging/postinst-check-hsa.sh b/packaging/postinst-check-hsa.sh new file mode 100755 index 00000000..9b0b2505 --- /dev/null +++ b/packaging/postinst-check-hsa.sh @@ -0,0 +1,50 @@ +#!/bin/sh +# Advisory check used as both the DEB postinst and the RPM %post scriptlet. +# TransferBench links against libhsa-runtime64.so.1; without it the binary +# fails at first run with a dynamic-linker error. The relocatable package +# declares no hard dep on hsa-rocr (or any ROCm component) because it is +# expected to install on TheRock-tarball systems where no ROCm package is +# tracked by apt/dpkg. Warn here so the user diagnoses a missing runtime +# at install time, not at TransferBench launch. +# +# Always exits 0 — this is advisory, never fatal. + +set -e + +found=0 + +if command -v ldconfig >/dev/null 2>&1; then + if ldconfig -p 2>/dev/null | grep -q 'libhsa-runtime64\.so\.1'; then + found=1 + fi +fi + +if [ "$found" -eq 0 ]; then + for d in /opt/rocm/lib /opt/rocm/lib64 /opt/rocm-*/lib /opt/rocm/extras-*/lib /opt/rocm/core-*/lib; do + if [ -e "$d/libhsa-runtime64.so.1" ]; then + found=1 + break + fi + done +fi + +if [ "$found" -eq 0 ]; then + cat >&2 <<'EOF' +==================================================================== +TransferBench: WARNING + +libhsa-runtime64.so.1 was not found on the dynamic loader path or +under any of /opt/rocm/lib, /opt/rocm-*/lib, or /opt/rocm/extras-*/lib. + +TransferBench requires the ROCm HSA runtime at run time. Install a +ROCm 7.x stack (system packages or TheRock SDK) before invoking +TransferBench, or set LD_LIBRARY_PATH to a directory containing +libhsa-runtime64.so.1. + +Without it, TransferBench will fail at startup with: + error while loading shared libraries: libhsa-runtime64.so.1 +==================================================================== +EOF +fi + +exit 0 diff --git a/src/client/Presets/AllToAll.hpp b/src/client/Presets/AllToAll.hpp index 16c3d1f6..cfea85c3 100644 --- a/src/client/Presets/AllToAll.hpp +++ b/src/client/Presets/AllToAll.hpp @@ -20,6 +20,8 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ +#include + int AllToAllPreset(EnvVars& ev, size_t const numBytesPerTransfer, std::string const presetName, diff --git a/src/client/Presets/AllToAllN.hpp b/src/client/Presets/AllToAllN.hpp index a67608e7..15698917 100644 --- a/src/client/Presets/AllToAllN.hpp +++ b/src/client/Presets/AllToAllN.hpp @@ -20,6 +20,7 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ +#include #include "EnvVars.hpp" int AllToAllRdmaPreset(EnvVars& ev, diff --git a/src/client/Presets/NicPeerToPeer.hpp b/src/client/Presets/NicPeerToPeer.hpp index dd94f468..ff67d8ac 100644 --- a/src/client/Presets/NicPeerToPeer.hpp +++ b/src/client/Presets/NicPeerToPeer.hpp @@ -20,6 +20,8 @@ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. */ +#include + // Helper functions int GetClosestDeviceToNic(MemType memType, int nicIdx, int rank) { diff --git a/src/header/TransferBench.hpp b/src/header/TransferBench.hpp index 5c16a784..3ff0d276 100644 --- a/src/header/TransferBench.hpp +++ b/src/header/TransferBench.hpp @@ -25,7 +25,6 @@ THE SOFTWARE. #include #include #include -#include #include #include #include