Use or learn from `dcgm` for rapids doctor?

[betatim]

A short note from a talk about training LLMs that I just attended. The speaker from huggingface mentioned https://github.com/NVIDIA/DCGM as a very useful way to debug/diagnose problems. Maybe useful for rapids doctor?

---

Just wanted to make a note, if you already know it/dismissed it feel free to close the issue

[ncclementi]

Taking a quick look I don't think it applies, it seems to be more directed to assessing health of entire data centers unless they use it in a different way. Can you expand more on how they use it?

Installation wise it relies on a building from source with docker, which will defeat the purpose of making the package easy to install and use.

[betatim]

They didn't elaborate really. I was thinking that it might be a source of ideas what information to display or what to automatically diagnose. I think integrating/depending on it makes life too complicated.