Emit health alerts for missing GPUs

[Matthias247]

BMM should emit health alerts if it detects that a host has less GPUs than it is intended to have.

Right now this feature is covered by SKU validation. However SKU validation only occurs during limited points in time during the host lifecycle. It also only acts on in-band GPU data, and not out-of-band data (redfish).

There are 2 ways to detect the problem out of band:

1. Parse the system event log to look for associated problems. This could happen within the hw-health service.
2. Compare the site exploration report for a host against the list of expected GPUs. That requires to enhance SKU validation to happen at more times within the host lifecycle. It also requires SKU validation to take into account the SiteExploration EndpointExplorationReport.

[yoks]

Health should be able to see all GPU via BMC, so it can technically can check log for all GPU related errors he see in BMC and alert on it.

[Matthias247]

the problem statements includes GPUs which for any reason don't show up on the BMC. For these, the expected manifest/inventory would be required.

[ajf]

@Matthias247 Is this pretty straightforward what to do or do we need a design doc for it?

[Matthias247]

I don't think its super hard, but there's certainly some thoughts that would need to go into each solution. I think it's ok to discuss the design here?

E.g. for

Parse the system event log to look for associated problems. This could happen within the hw-health service.

How do we intend to do the parsing. Is it a new module directly in hw-health that parses the received log stream? Or do we write the log to disk, and from there on a different logparser component checks the information.

Compare the site exploration report for a host against the list of expected GPUs.

Copy&paste from above: "It requires to enhance SKU validation to happen at more times within the host lifecycle. It also requires SKU validation to take into account the SiteExploration EndpointExplorationReport."

Yet another approach could be that hw-health loads the SKU data, and is able to perform the redfish checks based on that.

[github-actions]

This issue has been inactive for 60 days and will be closed soon.
Please comment to keep it open.

[github-actions]

This issue was automatically closed due to inactivity after being marked as stale.

If you believe this issue is still relevant:

• Reopen with a comment providing updated details or context
• Add the lifecycle/frozen label to prevent it from being closed again