Get access to wgpu hal

[kushalkolar]

I was looking into this a while ago and I think wgpu-native doesn't give access to the hal (yet?). Context: I was looking into using low level Vulkan to do video encode/decode and put that into a texture that wgpu can access via the hal.

Anyways 2am thought but I didn't want to forget it so posting here. Will followup with more info soon.

[Vipitis]

Also #782

There is a large update to wgpu-native happening that exposes a bunch of native only features like external textures and Vulkan memory etc. It will be exciting for all the interior opportunities. It might be a few more weeks to be ready and then released. But once that's available we need to improve our native extras a lot to map all of it.
Meant to write an issue with all the native only features we don't have support but could and the ones coming up. But busy with other stuff currently.

[kushalkolar]

Thanks, good to hear I was remembering correctly and not hallucinating 😂

[Vipitis]

If you have a specific usecase in mind with maybe some partial building blocks, I can see what changes will be needed to get a minimal prototype working for the wgpu mapping. Will be all kinds of jank with CI artifacts or local builds... But once you see it working, it might be all the motivation you need.

Maybe external texture, maybe Vulkan dma buf?

[kushalkolar]

We'd love to do video encode/decode with Vulkan and share that buffer or Texture with wgpu!

[Vipitis]

from what I can see there might be two ways to do this:

• dma buffer access: https://docs.rs/wgpu/latest/wgpu/struct.FeaturesWGPU.html#associatedconstant.VULKAN_EXTERNAL_MEMORY_DMA_BUF where multiple vulkan devices can access the same memory? - but this only works on linux currently.

• external texture, an actual webgpu feature where you can use images like a video stream as a sample only texture in webgpu... however the current implementation for wgpu is not working on linux or android (but windows with d3d12 and metal?). https://docs.rs/wgpu/latest/wgpu/struct.FeaturesWGPU.html#associatedconstant.EXTERNAL_TEXTURE

we don't have either in wgpu-native and hence wgpu-py just yet (perhaps it works in the browser with #753 but I didn't try yet). But they are both included in gfx-rs/wgpu-native#617 which we could experiment with as is or wait a few weeks for it to be merged and a new wgpu-native realse.

There was also some discussion about providing a wrapper for the vendor specific hardware APIs around video (and image) encoding/decoding so it can be done in hardware instead of compute shaders. But I think this was closed for out of scope... since the webgpu spec in the browser solves this via the external texture interoperability (although I am not sure how encoding is meant to be done here, maybe the canvas context can be encoded in the browser directly?).