Skip to content
BranchesTags

Repository files navigation

tenferro-benchmark

Benchmark suite for tenferro-rs, comparing tenferro against PyTorch, JAX, Julia, HPTT, cuTENSOR, and other reference backends on CPU and GPU workloads.

Latest Results

The tracked reports under result/ are the source of truth for benchmark numbers (they are never duplicated into this README). Each file is the latest report for one target_profile × suite_id pair; older results live in git history only.

Target profile Suite Report
mac-cpu (Apple Silicon, native) cpu/einsum result/mac-cpu/cpu/einsum.md
mac-cpu cpu/cpu_ops result/mac-cpu/cpu/cpu_ops.md
mac-cpu cpu/linalg_jvp_vjp result/mac-cpu/cpu/linalg_jvp_vjp.md
mac-cpu cpu/permutation result/mac-cpu/cpu/permutation.md
linux-cpu (Linux devcontainer; collected as amd-cpu) cpu/einsum result/linux-cpu/cpu/einsum.md
linux-cpu cpu/cpu_ops result/linux-cpu/cpu/cpu_ops.md
linux-cpu cpu/linalg_jvp_vjp result/linux-cpu/cpu/linalg_jvp_vjp.md
linux-cpu cpu/permutation result/linux-cpu/cpu/permutation.md
linux-cpu linalg JVP/JVP repro result/linux-cpu/cpu/linalg_jvp_jvp.md
nvidia-gpu (CUDA devcontainer) gpu/dense result/nvidia-gpu/gpu/dense.md
nvidia-gpu gpu/einsum result/nvidia-gpu/gpu/einsum.md
nvidia-gpu gpu/sparse result/nvidia-gpu/gpu/sparse.md
nvidia-gpu gpu/tensornetwork result/nvidia-gpu/gpu/tensornetwork.md
nvidia-gpu gpu/linalg_jvp_vjp result/nvidia-gpu/gpu/linalg_jvp_vjp.md
nvidia-gpu gpu/permutation result/nvidia-gpu/gpu/permutation.md

Raw runs (per-timestamp run.yaml + machine-readable outputs) are written to data/results/<target_profile>/<suite_id>/<timestamp>/; the tracked report in result/ is regenerated from the newest run. See docs/results.md for the full layout.

Prerequisites

All profiles:

  • Rust toolchain (cargo).
  • uv for the Python environment (uv sync creates .venv from pyproject.toml).
  • External checkouts under extern/ (tenferro-rs, strided-rs, and problem data), fetched by ./scripts/setup_extern_deps.sh. Note that extern/strided-rs is required for any cargo build in this repository (Cargo resolves optional path-dependency manifests even when their feature is disabled), so run the setup script before building anything.

For mac-cpu, run the "All profiles" commands directly on the host. For the devcontainer profiles (amd-cpu / linux-cpu / nvidia-gpu), run them inside the corresponding devcontainer (devcontainer exec ... bash -lc 'uv sync && ./scripts/setup_extern_deps.sh') — the setup script expects the container's OPENBLAS_ROOT / MKLROOT environment, and the container's .venv must be built with the container's wheels (see the GPU note below).

Profile-specific:

  • mac-cpu: runs natively (no Docker); tenferro uses Accelerate.
  • amd-cpu / linux-cpu: the devcontainer CLI and Docker; tenferro defaults to OpenBLAS, oneMKL is optional.
  • nvidia-gpu: the CUDA devcontainer under .devcontainer/cuda/.
  • cpu/permutation suite: Julia on PATH (e.g. juliaup or brew install julia) for the Julia backends; the repo Project.toml pulls in JSON.jl and Strided.jl via Pkg.instantiate. Without julia, those columns are skipped with a warning. For the HPTT column (present in the tracked latest reports), also install cmake plus a C++ toolchain (macOS: brew install cmake) and pass PERMUTATION_EXTRA_FEATURES=hptt, because the hptt Cargo feature builds the vendored HPTT C++ library.

Workflow guides per platform: macOS CPU · Linux CPU devcontainer · NVIDIA GPU devcontainer.

Running the Benchmarks

CPU einsum + ops (macOS, native)

scripts/run_all.sh [NUM_THREADS] runs the CPU einsum suite (tenferro trace/eager vs PyTorch/JAX) plus the CPU ops microbenchmarks (primal linalg, JVP/VJP, eager backward), and regenerates result/<target_profile>/cpu/{einsum,cpu_ops,linalg_jvp_vjp}.md. Each invocation writes one thread-count snapshot and overwrites those three latest reports, so run thread counts sequentially when you want both 1T and 4T data in raw runs; the tracked mac-cpu reports match a 4-thread collection:

uv sync
./scripts/setup_extern_deps.sh
BENCHMARK_TARGET_PROFILE=mac-cpu ./scripts/run_all.sh 1
BENCHMARK_TARGET_PROFILE=mac-cpu ./scripts/run_all.sh 4

To regenerate all four tracked result/mac-cpu/cpu/*.md reports in the same shape as the published latest (4T einsum/ops, plus permutation at 1T and 4T with HPTT), run sequentially:

uv sync
./scripts/setup_extern_deps.sh
# Julia on PATH; for HPTT: brew install cmake (and a C++ toolchain)
BENCHMARK_TARGET_PROFILE=mac-cpu ./scripts/run_all.sh 4
PERMUTATION_EXTRA_FEATURES=hptt \
BENCHMARK_TARGET_PROFILE=mac-cpu ./scripts/run_permutation.sh 1 4

Quick smoke (single small instance, one run, no warmup):

BENCHMARK_TARGET_PROFILE=mac-cpu \
BENCH_INSTANCE=bin_matmul_256 \
BENCH_RUNS=1 \
BENCH_WARMUPS=0 \
PUBLICATION_GATE_SUITE=small \
  ./scripts/run_all.sh 1

Useful environment variables: BENCH_INSTANCE (restrict to one einsum instance), BENCH_RUNS / BENCH_WARMUPS (iteration counts), TENFERRO_CPU_FEATURES (BLAS provider: system-accelerate, system-openblas, system-mkl; macOS defaults to system-accelerate), RUN_PERMUTATION_SUITE=1 (also run the cpu/permutation suite after everything else, sequentially; HPTT still needs PERMUTATION_EXTRA_FEATURES=hptt).

Note: a full or smoke run_all.sh invocation overwrites the tracked latest reports under result/<target_profile>/. If you only ran a smoke subset, restore them before committing (git checkout -- result/).

CPU einsum + ops (Linux devcontainer)

Same runner, executed inside the devcontainer from the host:

devcontainer up --workspace-folder .
devcontainer exec --workspace-folder . bash -lc '
  BENCHMARK_TARGET_PROFILE=amd-cpu ./scripts/run_all.sh 1'

CPU permutation suite (cpu/permutation)

A standalone materialize/copy-kernel benchmark comparing tenferro-rs to_contiguous against strided-rs, HPTT, Julia Base, and Strided.jl. An internal untimed odometer implementation provides the correctness reference. Every timed call includes fresh destination allocation for a common end-to-end materialization comparison. Spec: docs/permutation-suite.md.

uv sync
./scripts/setup_extern_deps.sh
# Multiple thread counts are measured sequentially in one run.
# Without PERMUTATION_EXTRA_FEATURES=hptt the HPTT column is omitted (`-`);
# without `julia` on PATH the Julia columns are omitted. Tracked latest
# reports include both.
PERMUTATION_EXTRA_FEATURES=hptt \
BENCHMARK_TARGET_PROFILE=mac-cpu ./scripts/run_permutation.sh 1 4

For a quick trial run, the suite honors PATTERN_ID (restrict to one pattern), BENCH_RUNS, and BENCH_WARMUPS:

BENCHMARK_TARGET_PROFILE=mac-cpu \
PATTERN_ID=transpose_2d_2048 \
BENCH_RUNS=1 \
BENCH_WARMUPS=0 \
  ./scripts/run_permutation.sh 1

This writes result/<target_profile>/cpu/permutation.md. Pattern definitions live in data/instances/permutation_patterns.json and are read by both the Rust and Julia runners; result records are validated against schemas/permutation-result.schema.json. The suite can also be appended to a run_all.sh invocation with RUN_PERMUTATION_SUITE=1.

GPU suites (CUDA devcontainer)

The repo .venv is shared between the CPU and CUDA devcontainers (it lives in the bind-mounted workspace). A CPU-side uv sync — including the CPU devcontainer's own post-create hook — replaces the CUDA wheels with CPU ones. The GPU Python runners then silently skip pytorch-cuda / jax-cuda (they exit 0 and the report is simply missing those columns) rather than failing. Before collecting GPU results, install the CUDA Python backends inside the container and verify they see the GPU:

devcontainer up --workspace-folder . --config .devcontainer/cuda/devcontainer.json

# One-time, and again after ANY CPU-side `uv sync`:
devcontainer exec --workspace-folder . --config .devcontainer/cuda/devcontainer.json \
  bash -lc '
    (uv sync --frozen || uv sync)
    uv pip install "torch>=2.12.0" --extra-index-url https://download.pytorch.org/whl/cu126
    uv pip install "jax[cuda12]"
    ./scripts/setup_extern_deps.sh
    uv run python -c "import torch, jax; assert torch.cuda.is_available(); jax.devices(\"cuda\"); print(\"CUDA OK:\", torch.__version__)"'

If nvidia-smi fails inside a previously created container ("Failed to initialize NVML" / CUDA_ERROR_NO_DEVICE) while the host GPU is fine, docker restart <container> usually restores GPU access — no rebuild needed.

# gpu/dense, gpu/einsum, gpu/sparse, gpu/tensornetwork:
devcontainer exec --workspace-folder . --config .devcontainer/cuda/devcontainer.json \
  bash -lc 'BENCHMARK_TARGET_PROFILE=nvidia-gpu ./scripts/run_gpu_suite.sh'

# gpu/linalg_jvp_vjp (separate from the standard GPU suite):
devcontainer exec --workspace-folder . --config .devcontainer/cuda/devcontainer.json \
  bash -lc 'BENCHMARK_TARGET_PROFILE=nvidia-gpu ./scripts/run_gpu_linalg_jvp_vjp.sh'

# gpu/permutation (standalone, like cpu/permutation):
devcontainer exec --workspace-folder . --config .devcontainer/cuda/devcontainer.json \
  bash -lc 'BENCHMARK_TARGET_PROFILE=nvidia-gpu ./scripts/run_gpu_permutation.sh'

gpu/permutation honors the same quick-trial variables as the CPU suite (PATTERN_ID, BENCH_RUNS, BENCH_WARMUPS), plus GPU_BENCH_DEVICE for the CUDA ordinal:

devcontainer exec --workspace-folder . --config .devcontainer/cuda/devcontainer.json \
  bash -lc 'BENCHMARK_TARGET_PROFILE=nvidia-gpu \
    PATTERN_ID=transpose_2d_2048 BENCH_RUNS=1 BENCH_WARMUPS=0 \
    ./scripts/run_gpu_permutation.sh'

After any GPU run, check the generated report for the full backend set (the Comparison Backends section lists what each suite compares; for gpu/permutation that is two tenferro columns, cuTENSOR, PyTorch CUDA, JAX CUDA, and memcpy-d2d). A column that is - on every row means that backend's runner skipped — usually the CUDA-wheels issue above. Like run_all.sh, these scripts overwrite the tracked latest reports under result/nvidia-gpu/; after a trial or partial run, restore them before committing (git checkout -- result/).

The GPU tensor network benchmark uses problem data from extern/TensorNetworkBenchmarks/, based on the upstream TensorNetworkBenchmarks repository; see docs/tensornetwork-gpu.md.

Linux linalg AD repro (OpenBLAS / oneMKL)

Reproduces result/linux-cpu/cpu/linalg_jvp_jvp.md with the devcontainer's source-built OpenBLAS (/opt/openblas):

devcontainer up --workspace-folder . --remove-existing-container
devcontainer exec --workspace-folder . bash -lc '
  python3 - <<PY
import ctypes
lib = ctypes.CDLL("/opt/openblas/lib/libopenblas.so")
lib.openblas_get_config.restype = ctypes.c_char_p
lib.openblas_get_parallel.restype = ctypes.c_int
print(lib.openblas_get_config().decode())
print(f"parallel={lib.openblas_get_parallel()}")
PY'
devcontainer exec --workspace-folder . bash -lc '
  export TENFERRO_CPU_FEATURES=system-openblas
  export PUBLICATION_GATE_FEATURES=system-openblas
  export TENFERRO_CPU_BACKEND_KIND=blas
  ./scripts/reproduce_linux_cpu_linalg_jvp_jvp.sh'

For the oneMKL variant (/opt/intel/oneapi/mkl/latest), replace both system-openblas values with system-mkl. Verify the OpenBLAS build through the runtime API above instead of relying on strings.

Measurement Policy

  • Benchmarks are always run sequentially — never multiple benchmark processes at once, including different thread-count variants of the same suite (see AGENTS.md).
  • Thread counts are controlled via RAYON_NUM_THREADS / OMP_NUM_THREADS / JULIA_NUM_THREADS. CPU pinning is unavailable on macOS; on Linux the devcontainer convention applies no taskset/numactl pinning either. The effective thread environment is recorded in each run's run.yaml.
  • Every run records provenance (target profile, suite, tenferro-rs commit, CPU/GPU info) in data/results/.../run.yaml.

Comparison Backends

CPU einsum reports compare tenferro-trace, tenferro-eager, pytorch-cpu, and jax-cpu. GPU reports compare tenferro-cuda-trace, tenferro-cuda-eager, pytorch-cuda, jax-cuda, and vendor-specific CUDA backends where meaningful. The cpu/permutation suite has its own backend set (tenferro-rs to_contiguous, HPTT, strided-rs, Julia Base, Strided.jl, memcpy); gpu/permutation compares tenferro CUDA transpose paths against cuTENSOR, PyTorch/JAX CUDA, and a device-to-device memcpy baseline.

C++ Torch/LibTorch runners are intentionally removed; PyTorch Python is the ATen comparison backend. The PyTorch CPU provider is detected at run time and recorded in run.yaml and generated reports; Linux does not source-build PyTorch to force OpenBLAS.

Documentation

Development Checks

Run these after changing benchmark scripts or schemas:

uv run python scripts/validate_benchmark_suite.py benchmarks/cpu/einsum.yaml benchmarks/cpu/permutation.yaml
uv run python scripts/validate_benchmark_suite.py benchmarks/gpu/dense.yaml benchmarks/gpu/einsum.yaml benchmarks/gpu/sparse.yaml benchmarks/gpu/tensornetwork.yaml
bash tests/test_suite_result_layout.sh
bash tests/test_run_all_docs_outputs.sh
bash tests/test_clean_extern_deps.sh
bash tests/test_setup_extern_tenferro_checkout.sh
bash tests/test_permutation_result_schema.sh
cmake -S cpp -B build/cpp-plan-test
cmake --build build/cpp-plan-test --target einsum_plan_test
ctest --test-dir build/cpp-plan-test --output-on-failure

License

MIT

About

tenferro benchmark suite

Resources

Readme
Activity
Custom properties

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages