Topic/drm amdgpu kfd pin reaper

[chun-wan]

Introduce a new KFD-side subsystem that lets multi-tenant HIP/AMDGPU
serving workloads opt the free / unpin paths into bounded-wait failure
modes instead of either silently orphaning RDMA-pinned BOs or
deadlocking on a wedged peer driver. All five new module parameters
default to 0 (feature off); the resulting code paths are byte-identical
to the existing upstream behaviour when none are set.

amdgpu_kfd_free_wait_ms max ms to bounded-wait on pin_count==0
in amdgpu_amdkfd_gpuvm_free_memory_of_gpu()
when hipFree() hits an RDMA-pinned BO

amdgpu_kfd_unpin_drain_ms strict dma_resv_wait_timeout deadline
in the unpin path -- prevents an
in-flight peer-RDMA WR from racing the
pin_count drop and faulting the peer

amdgpu_kfd_free_on_pinned policy switch (0 = legacy orphan,
1 = bounded wait + queue for reaper)

amdgpu_pin_orphan_timeout_ms age (ms) at which the reaper force-
unpins a BO parked on the orphan list

amdgpu_pin_reaper_interval_ms tick interval (ms) of the reaper
background delayed_work

Motivation

In multi-tenant HIP serving workloads, RDMA peer drivers (e.g. the
NIC-side rdma_pin or peer-mem shim) hold pin_count > 0 on KFD BOs
across the lifetime of in-flight RDMA WRs. When userspace calls
hipFree() on such a BO upstream amdgpu either:

1. silently keeps the BO alive until process exit (memory leak
   that accumulates over hours of serving), or
2. blocks the unpin path indefinitely if the peer driver is
   wedged (one stuck NIC turns into a process-wide free-path stall).

The five knobs introduced here let operators opt the KFD layer into a
controlled non-leak / non-deadlock policy:

• hipFree() bounded-waits up to amdgpu_kfd_free_wait_ms for the peer
  to drop the pin (typical: 2-4 seconds);
• if the peer is still holding, the BO is queued for the orphan
  reaper, which periodically (amdgpu_pin_reaper_interval_ms) walks
  the list and force-unpins entries older than
  amdgpu_pin_orphan_timeout_ms;
• the unpin path itself uses amdgpu_kfd_unpin_drain_ms to bound the
  dma_resv_wait_timeout() that drains in-flight fences before
  pin_count is decremented, so a wedged peer cannot park the unpin
  caller indefinitely.

Technical Details

NEW FILE

• drivers/gpu/drm/amd/amdgpu/amdgpu_kfd_pin_reaper.c
  amdgpu_kfd_unpin_drain() bounded dma_resv_wait_timeout()
  amdgpu_kfd_wait_pin_drop() exp-backoff poll on pin_count==0
  amdgpu_kfd_orphan_queue() queue BO onto adev->kfd.orphan_list
  amdgpu_kfd_reaper_fn() delayed_work callback that walks the
  orphan list and force-unpins aged
  entries
  amdgpu_kfd_reaper_start() one-shot init from device_init
  amdgpu_kfd_reaper_stop() one-shot teardown from device_fini

MODIFIED

• drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.h
  Add struct kfd_orphan_pin (queue node).
  Add 9 new fields to struct amdgpu_kfd_dev: orphan list spinlock,
  list head, reaper delayed_work, atomic64 stat counters
  (rdma_pin_orphans_queued/reaped, free_wait_pinned_count/timeout,
  unpin_drain_timeouts), reaper_started bool.
  Add function declarations for the helpers + reaper lifecycle.

• drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c
  amdgpu_amdkfd_device_init() -> amdgpu_kfd_reaper_start(adev)
amdgpu_amdkfd_device_fini_sw() -> amdgpu_kfd_reaper_stop(adev)
  (reaper_stop is called first so the worker exits before
  adev->kfd.dev is torn down).

• drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd_gpuvm.c
  amdgpu_amdkfd_gpuvm_free_memory_of_gpu() gains a small policy
  block that runs only when mem->bo->tbo.pin_count > 0 AND
  amdgpu_kfd_free_on_pinned >= 1. When the bounded wait times
  out the BO is queued via amdgpu_kfd_orphan_queue() and
  hipFree() returns success to userspace; the BO is reclaimed
  either when the peer releases the pin or forcibly by the reaper.

• drivers/gpu/drm/amd/amdgpu/amdgpu_drv.c
  Five new module_param_named() declarations with DOC comments.

• drivers/gpu/drm/amd/amdgpu/amdgpu.h
  Five new extern int declarations.

• drivers/gpu/drm/amd/amdgpu/Makefile
  Add amdgpu_kfd_pin_reaper.o to amdgpu-y.

Default behaviour

All five module parameters default to 0. When none are set:

• amdgpu_kfd_reaper_start() initialises the orphan list and atomic
  counters but does NOT schedule the reaper delayed_work (because
  amdgpu_pin_reaper_interval_ms == 0); the worker never runs.
• The free-on-pinned policy block in free_memory_of_gpu() is gated
  by amdgpu_kfd_free_on_pinned >= 1, so it is a no-op.
• amdgpu_kfd_unpin_drain() returns 0 immediately when its timeout
  argument is 0.

Net behavioural diff on stock callers: zero.

JIRA ID

ROCM-21571

Test Plan

Reproducer: multi-tenant HIP serving with RDMA peer driver holding
pin_count across hipFree(). Open-source reproducer to be added in a
follow-up test patch.

Expected behaviour:

• All five envs unset (default): unchanged. hipFree() on a pinned
  BO leaks (legacy) as before.
• amdgpu_kfd_free_on_pinned=1, kfd_free_wait_ms=4000,
  pin_orphan_timeout_ms=10000, pin_reaper_interval_ms=2000:
  hipFree on a pinned BO either waits up to 4 s for the peer to
  release (and then frees cleanly), or queues the BO for the
  reaper. The reaper ticks every 2 s and force-unpins anything
  older than 10 s.

Test Result

11+ hours of multi-tenant HIP serving load on a customer MI300X
stable with the five envs set per the recipe above; orphan reaper
queued / reaped counters match expectations (queued == reaped over
the run), and no memory leak observed across 32-cycle stress
sweep. Default-off run passes existing KFD self-tests
byte-identically.

Submission Checklist

• Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

Signed-off-by: chun-wan chun-wan@users.noreply.github.com