Multi-GPU Pipeline Parallel

[rtlinux]

Hello team, I have two A770 and really enjoy the OpenARC PP performance!

According to the documentation we could use

--device HETERO:GPU.0,GPU.1 \ --runtime-config "{"MODEL_DISTRIBUTION_POLICY": "PIPELINE_PARALLEL"}"

Has anyone actually got this to work? Do we need an oder version of OpenVINO or need to compile it from source?
openvinotoolkit/openvino@master...ambeckley:openvino:fix/hetero-multi-gpu-split-ratio

Thanks for your help!

[SearchSavior]

@rtlinux Hey! @ambeckley fixes DO work with int8 only.

There is a build process, we have a doc on discord which covers the build flow for building openvino, and then openvino genai + openvino tokenizers. The process is not hardened yet, and OpenArc cannot support atm. However I am working on implementing us a vllmish backend to support, and in the future I think we will make a build script which patches openvino with his fixes. The parts of openvino ambeckleys code touches are not actively maintained;

I have been trying in issues to get his fixes escalated; the best case is intel supports and eventually patches int4 quantization so we can run larger models.

there is an additional fix needed for openvino genai I added in a comment here: openvinotoolkit/openvino#34702 (comment)

Please leave this issue open!

[t8]

Hi guys, a bit late but just wanted to share our team has been working on something for pipeline-parallel inference on Intel hardware. Perhaps this could help? https://github.com/labscommunity/cascadia