feat: add Enflame GCU TOPSPTI profiler integration

[lvyufeng]

AI Agent Information

• Agent/Tool: Claude Code CLI
• Model: Claude Opus 5 (1M context)
• Human Reviewer: @lvyufeng
• Session Summary: Synchronized the GCU feature branch with the latest upstream history and implemented optional Enflame GCU TOPSPTI profiler support. Hardware validation remains pending because this environment has no Torch-enabled build or physical GCU device.

Summary

This PR adds an optional TOPSPTI-backed DeviceTracer for Enflame GCU and connects it to the existing vendor-neutral Kineto profiler adapter. The implementation decodes kernel, memcpy, memset, runtime, and driver activities, calibrates TOPSPTI timestamps to realtime, and preserves vendor correlation flow data plus best-effort Kineto external correlation. It also adds CMake SDK detection, unavailable fallbacks, a GCU integration test, and architecture documentation.

Change Type

• Bug Fix
• New Feature
• Performance Optimization
• Refactoring
• Documentation
• Testing
• CI/Infrastructure
• Breaking Change

Platforms Affected

• CUDA
• MetaX
• Ascend
• PPU
• Enflame GCU
• Platform-agnostic (all platforms)

Problem Analysis

What was broken/missing?

The GCU backend had no implementation of the profiler DeviceTracer contract. A Kineto session could not collect GCU device activities, runtime-to-device flow relationships, or GCU-specific activity metadata through the existing PrivateUse1 profiler path.

Why did it happen?

The repository already had vendor-specific tracers for CUDA-compatible and other accelerator runtimes, but Enflame's TOPSPTI activity API had not been adapted to the shared DeviceTracer abstraction. TOPSPTI also does not expose the CUPTI-style external-correlation push/pop API, so its correlation behavior requires a platform-specific mapping strategy.

Investigation process:

1. Inspected csrc/profiler/device_tracer.h, the generic Kineto adapter, and existing CUDA, MSPTI, and MUPTI tracer implementations.
2. Inspected the installed Enflame SDK headers and libraries under /opt/tops, identifying the asynchronous activity API in topspti_activity.h and libtopspti.so as the appropriate online tracing interface.
3. Rebased the branch onto the latest flagos/main, resolved the RNG and operator-support conflicts while retaining both local and upstream changes, and checked that GCU source selection leaves exactly one tracer factory.
4. Compiled the tracer against TOPSPTI headers, against a GCU configuration without the SDK headers, and in the non-GCU fallback configuration.

Solution Design

Implementation approach:

• Add gcu_topspti_device_tracer.cc implementing the shared DeviceTracer interface.
• Add topspti_shim.h with optional header declarations and lazy dlopen/dlsym binding.
• Detect TOPSPTI headers in CMake and define FLAGOS_HAVE_TOPSPTI only when available.
• Enable TOPSPTI activity kinds only for an active Kineto session and release asynchronous buffers through the vendor callback.
• Convert TOPSPTI records into generic events with timestamps, names, device/stream/thread fields, and metadata.
• Register the generic Kineto profiler when TOPSPTI support is compiled in.
• Add a GCU integration test that exports Chrome trace JSON and checks kernel/runtime event validity.

Key design decisions:

• Lazy runtime loading: The profiler library is loaded only when a profiling session starts. This avoids imposing a TOPSPTI runtime dependency on ordinary GCU operator processes.
• Graceful fallback: Missing headers or an unavailable runtime produces an unavailable tracer instead of a build or import failure.
• Generic abstraction boundary: TOPSPTI types remain confined to the GCU tracer and shim; Kineto continues to consume DeviceEvent values.
• Clock calibration: TOPSPTI timestamps are mapped to realtime using start/stop samples of topsptiGetTimestamp() and CLOCK_REALTIME.
• Correlation handling: Vendor correlation IDs continue to pair runtime and device activity. Since TOPSPTI has no external-correlation API, the API-enter callback records the current Kineto correlation for later External id attribution when callback ordering supports it.

Code changes by file:

• csrc/CMakeLists.txt: Select the GCU tracer exclusively and detect optional TOPSPTI headers.
• csrc/profiler/flagos_kineto_profiler.cc: Register the generic Kineto adapter for FLAGOS_HAVE_TOPSPTI builds.
• csrc/profiler/gcu_topspti_device_tracer.cc: Decode TOPSPTI activity buffers, calibrate timestamps, map correlations, and implement the GCU tracer factory.
• csrc/profiler/topspti_shim.h: Provide lazy runtime symbol binding and no-SDK fallback declarations.
• tests/integration/conftest.py: Register the gcu pytest marker.
• tests/integration/test_profiler_gcu.py: Add physical-GCU TOPSPTI activity and Chrome trace assertions.
• docs/architecture/profiler.md: Document the GCU integration, correlation limitations, and current validation status.

Changes by commit:

1. 083c616 - feat: add GCU TOPSPTI profiler integration: Add the tracer, runtime shim, build wiring, test, and documentation on top of the latest upstream main.

Verification

Pre-submission Checklist

• Linting passed for changed Python files (ruff check and ruff format --check)
• Type checking passed (not applicable)
• All tests pass (blocked: pytest is unavailable in this environment)
• Manual hardware testing completed (blocked: no physical GCU device)
• No debug/temporary code (diagnostic logging is opt-in through FLAGOS_TOPSPTI_DEBUG)
• Documentation updated
• Commit messages follow conventions
• All text in English

Linting Results

Targeted checks for the changed Python files:

$ ruff check tests/integration/conftest.py tests/integration/test_profiler_gcu.py
All checks passed!

$ ruff format --check tests/integration/conftest.py tests/integration/test_profiler_gcu.py
2 files already formatted

Repository-wide checks were also attempted:

$ ruff check
# Failed on pre-existing violations in unrelated benchmark/scripts files.
# No changed profiler Python file was reported after the targeted check.

$ ruff format --check
# Failed because 12 pre-existing files would be reformatted, including documentation
# and skill files unrelated to this PR.

Test Results

$ pytest tests/unit/ -v
/bin/bash: pytest: command not found

$ pytest tests/integration/test_profiler_gcu.py -v
/bin/bash: pytest: command not found

The GCU integration test therefore has not been run. It is designed to skip when ACCELERATOR=gcu, the FlagOS backend, or the TOPSPTI runtime is unavailable, and requires physical GCU validation before this PR is marked ready for review.

Manual Verification

The C++ source-level checks completed successfully. The only diagnostic is the existing unused-parameter warning from DeviceTracer::pushCorrelation in device_tracer.h:

$ g++ -std=c++17 -Wall -Wextra -fsyntax-only \
    -DUSE_GCU -DFLAGOS_HAVE_TOPSPTI \
    -Icsrc/profiler -I/opt/tops/extras/TOPSPTI/include \
    csrc/profiler/gcu_topspti_device_tracer.cc
# Exit code: 0
# Warning: existing unused parameter `id` in device_tracer.h

$ g++ -std=c++17 -Wall -Wextra -fsyntax-only \
    -DUSE_GCU -Icsrc/profiler \
    csrc/profiler/gcu_topspti_device_tracer.cc
# Exit code: 0

$ g++ -std=c++17 -Wall -Wextra -fsyntax-only \
    -Icsrc/profiler \
    csrc/profiler/gcu_topspti_device_tracer.cc
# Exit code: 0

$ git diff --check
# No output; passed

A complete CMake configure/build was attempted but is blocked because the active Python environment has no PyTorch installation or Torch CMake package:

Could not find a package configuration file provided by "Torch"
ModuleNotFoundError: No module named 'torch'

Code Quality Verification

Style Consistency

• Matched existing code style in modified files
• Followed naming conventions and existing DeviceTracer patterns
• Kept comments focused on vendor/API limitations
• Reused the repository's generic Kineto adapter and tracer contract

Edge Cases Considered

1. TOPSPTI headers are absent at build time.
2. TOPSPTI runtime symbols or libraries are absent at runtime.
3. Activity buffers fail allocation or contain malformed records.
4. Activity timestamps are zero, reversed, or implausibly long.
5. Runtime callbacks are unavailable or do not arrive before activity draining; device events still remain available, while external correlation may be absent.
6. The vendor device count API is unavailable; the tracer falls back to one device.

Potential Risks

1. TOPSPTI callback ordering and correlation semantics have not been validated on physical GCU hardware, so External id attribution may require adjustment for a particular SDK release.
2. TOPSPTI structure layouts and library symbol availability may vary across TopsRider SDK releases.
3. The current environment did not permit a complete Torch build or integration run.

Rollback Plan

Revert commit 63f8808. This removes the GCU TOPSPTI tracer, CMake detection, test, registration condition, and documentation without changing the existing CUDA, MetaX, Ascend, or MUSA tracer implementations.

Related Work

• Related to the existing generic Kineto profiler adapter and the MUSA MUPTI integration.
• Builds on the existing Ascend MSPTI profiler integration.
• No issue number was provided for this implementation.

Explicitly Not Included

• Physical Enflame GCU hardware validation and captured Chrome trace evidence.
• Full Torch-enabled CMake build verification in this environment.
• Changes to unrelated repository-wide Ruff violations.
• Vendor-native operator integration or generated operator support changes.

Human Review Notes

Areas needing special attention:

1. Review TOPSPTI activity structure decoding and compatibility with supported TopsRider SDK versions.
2. Validate API-enter callback ordering and confirm runtime/device correlation plus External id attribution on physical GCU hardware.
3. Review asynchronous buffer lifetime and session start/stop behavior under concurrent GCU work.

Questions for reviewer:

1. Which TopsRider SDK and GCU model should be used for the first hardware validation run?
2. Does the target SDK expose a more reliable external-correlation mechanism or callback ordering guarantee that should replace the current best-effort mapping?

Additional Context

The installed SDK was inspected under /opt/tops; the implementation uses the public asynchronous activity API from /opt/tops/extras/TOPSPTI/include and resolves libtopspti.so lazily. Runtime lookup can be overridden with FLAGOS_TOPSPTI_LIBRARY, and diagnostics can be enabled with FLAGOS_TOPSPTI_DEBUG=1.

---

🤖 Generated with Claude Code