Add back GPU tests to CI pipeline

[joshbainbridge]

GPU tests were removed from the CI pipeline as the self-hosted runner hardware (Jetson Nano) became non-supported by GitHub and its OS requirements.

This testing is useful to validate that the GPU implementation builds and the results exactly match that of CPU implementations. Enabling this will mean sourcing a runner that supports CUDA architecture.

[jfpanisset]

The following ASWF projects demonstrate a GPU test suite:

• OCIO uses AWS CodeBuild to run its OpenGL testsuite using EGL for window management, allowing its tests to run headless and not requiring an X server
• OSL uses for pay GitHub T4 GPU runners which ASWF projects have access to. OSL uses OptiX and does not need an X server
• fvdb uses custom, self managed, "spun up on demand" AWS EC2 instances

Feel free to ping the #wg-ci channel on ASWF Slack to discuss available options and what would work best for OpenQMC.

[joshbainbridge]

Hi @jfpanisset! Thank you for the guidance on this, that is super helpful. I'll definitely make sure I'm on the WG.

OpenQMC only needs access to CUDA compute, but does not need OpenGL or anything display-oriented. It can also be a very minimal system. Sounds like what OSL currently has would suit our needs. But it would be good to get your opinion on that if possible.

A quick overview is that our CI compiles some example projects using the library, and then executes them as CLI tools, all very simple stuff. OpenQMC supports running in CUDA, and so previously we had GPU-based variants of these example projects build and tested against the CPU output, checking for identical results.

This was all disabled some time ago due to the loss of the self-hosted hardware we were using, and as a result we haven't had that live testing on GPU since.

Another interesting point here is that since then, our CPU tests have moved to Nix to avoid constant churn on GHA. I don't think this will be possible with GPUs due to the external driver dependencies, so we might need to find a setup for that as a corner case.

[jfpanisset]

OSL would def be the right place to start then for "compute only" GPU CI usage (OSL does use OptiX but you can ignore that part and just use CUDA).

I am planning on creating specific ci-openqmc container images with the dependencies required to build OpenQMC, in the meantime you can look at the existing set of what's currently being built and figure out what's closest to the dependencies you need.

https://hub.docker.com/u/aswf

You could just use ci-vfxall since that has every project built by aswf-docker:

https://github.com/AcademySoftwareFoundation/aswf-docker/tree/main/ci-vfxall

Note that those README.md files are not complete, as they don't show the transitive dependencies of those packages, I'm planning to fix this.

Do you have a list of dependencies that OpenQMC requires to build? I could easily tell you the closest match. All of the ci-XXXX images have CUDA, OptIX and OpenGL in them, so any of them can be used to run CUDA workloads.

Also since GPU runners are "for pay", they will only run for GHA jobs in the context of the AcademySoftwareFoundation GitHub org, not in the context of a pre-merge check on a PR. So GPU testing would best run on a nightly / weekly basis, or perhaps on a post-PR merge trigger.