Not supported on CPU

[wbrao]

Inference is not supported on the CPU. Is the MSDA module implemented purely in CUDA?

[jerpelhan]

Inference on CPU was not tested, as the model has only been validated on GPU. Although CPU kernels for MSDA are implemented, it could be that some simple modifications would be needed.

[jerpelhan]

May I ask what error you get if you run demo_gradio.py so it loads on CPU rather than GPU?

[wbrao]

First, after I modified the device to run on the CPU, some attributes in C_base were missing due to conditional checks

After fixing the above problem, MSDA reported an error that CPU is not supported. The error message and modifications are as follows. Can this issue be resolved by replacing MSDA with a version that can run on the CPU?

Source code:
class MSDeformAttnFunction(Function):
    @staticmethod
    def forward(ctx, value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights, im2col_step):
        ctx.im2col_step = im2col_step
        output = MSDA.ms_deform_attn_forward(
            value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights, ctx.im2col_step)
        ctx.save_for_backward(value, value_spatial_shapes, value_level_start_index, sampling_locations, attention_weights)
        return output 

Error info：
RuntimeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
output = MSDA.ms_deform_attn_forward(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Not implemented on the CPU

[jerpelhan]

Yes, exactly, and there is already a CPU-compatible MSDA included in the repository; it was simply not wired to be called, since I did not intend for this model to run on CPU.

Additionally, it is probably best to omit the prediction of sam_masks (to speed up prediction) in counter_infer.py by introducing an extra flag (e.g., return_masks) and adjusting the code in lines 147–161 accordingly.

Sorry about that... I would fix it myself, but I am currently extremely busy with other projects. Please feel free to open a pull request with the fix 👍

[wbrao]

Thank you very much for your explanation. I would also like to ask specifically where this CPU-compatible MSDA is provided in the source code repository.

[jerpelhan]

The CPU-compatible version is already present in the repository, but it is **not in the C++ operator, it is the PyTorch reference implementation.

You can find it here: models/ops/functions/ms_deform_attn_func.py , specifically, the function: ms_deform_attn_core_pytorch().

This is a pure PyTorch implementation, which should be equivalent but was never tested by me.