feat: Per-Object (Machine/Instance/Host) State Progress Observability

[fengzhou-nv]

Is this a new feature, an enhancement, or a change to existing functionality?

Enhancement

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

Infra-controller currently exposes aggregate state-controller metrics, state history, and transition counters. However, they do not contain enough information, such as

• which machines require manual operation action (in an easily queryable form for large scale environment)
• custom sla for different machines (e.g. different hardware generation), configuration, or other factors (e.g. machines used by different tenants)
• the current state of the machine (object) as metric, which can be used to join with other metrics (e.g. DPU side), one use case is that it does not make sense to alert DPU not calling home (dpu-agent not communicating with api or sending logs) if the machine is in dpu reprovision state

Infra-controller provides Prometheus metrics and transition events that allow SRE-owned Prometheus rules to detect machines or DPUs stuck in a state, measure how long they have been stuck, show the expected next transition, and mark cases requiring manual operator action.

Feature Description

infr-controller provides per-object (machine, instance or dpu id) metrics for

• life cycle timestamps, e.g. time entered the state
• state machine metrics (e.g. current state, substate of the object)
• state transition metrics such as transition status, transition timing (note: per-object, not aggregate)
• whether manual operation is required (state machine has no way to resolve or give up)
• other key measures related to the machine (e.g. network config sync lag). There are typically dependency operations of some state transition steps (e.g. network config sync, infiniband sync for instance creation)
• stable metrics w.r.t. the object's physical traits (e.g. the machine's vendor, machine's rack, etc.)
• metric recording commonly used associations with other objects (e.g. host <-> dpu) which can be use to perform joins in Prometheus queries

If the stuck (manual operation) is due to some hardware health issues, expose the metrics to indicate such (even greater is to expose the hardware, e.g. p0_if cable, that's causing the issue).

The intended use of these metrics are define alerting rules such as

carbide_state_age_seconds{machine_id!~'...'} > carbide_state_sla_warning_seconds
carbide_state_age_seconds{machine_id!~'...'} > carbide_state_sla_critical_seconds
count(carbide_manual_intervention_required{...}) / carbide_total_machines > 5%
histogram_avg(network_config_sync_lag_seconds{..}[10m]) > 600
join(current_timestamp - carbide_forge_dpu_agent_last_call{dpu_id}) on dpu_id (machines{dpu_id, machine_id})) unless carbide_current_state{machine_id} == 'Reprovisioning'

I am not sure if it's feasible to also expose next expected state for each object as metrics, given some times object may transition to different states based on configurations/conditions. But if feasible, it would be great to have them in metrics as well.

Here are what the new metrics could look like (for illustrative purpose, does not need to be exactly like this):

{{carbide_state_age_seconds{
object_type="machine",
machine_id="...",
dpu_id="...",
normalized_state="host_init",
normalized_substate="waiting_for_discovery"
}}}

 

{{carbide_state_entered_timestamp_seconds{
object_type="machine",
machine_id="...",
normalized_state="host_init",
normalized_substate="waiting_for_discovery"
}}}

 

{{carbide_state_expected_next_transition{
object_type="machine",
machine_id="...",
normalized_state="host_init",
normalized_substate="waiting_for_discovery",
expected_next_state="validation",
expected_next_substate="machine_validation"
}}}

 

{{carbide_state_transitions_total{
object_type="machine",
from_state="host_init",
from_substate="waiting_for_discovery",
to_state="validation",
to_substate="machine_validation",
result="success|failed"
}}}

 

{{carbide_manual_intervention_required{
object_type="machine",
machine_id="...",
normalized_state="host_reprovision",
normalized_substate="waiting_for_firmware_upgrade",
reason="firmware_task_stalled"
}}}

 

{{carbide_network_config_sync_lag_seconds{
machine_id="...",
dpu_id="..."
}}}

The motivation of this ask is that: imagine a cluster with 100k machines, that has 5% having issues (2% of which were out-of-sla in instance creation caused by slowness in network config sync, 2% were caused by dpu oob issues, 1% caused by temperature sensor issues). How long does it take to triage the 5% (5k machines) error, asses the impact and organize incident responses for each cause? Going through each machine in the web ui will be impossible at this scale.

Describe your ideal solution

No response

Describe any alternatives you have considered

No response

Additional context

No response

Code of Conduct

• I agree to follow NCX Infra Controller's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[dathan]

This would be helpful to understand where we run into the biggest blockers in lifecycle of the rack, dpu, host what have you.

[Matthias247]

related: #191, #2168

[yoks]

I would make it a epic and create necessary metrics one by one #191 would be first task on it. If it works we can add more.

[yoks]

#2168 builds base to support this in future PR.

[mxh-0xbb]

I suggest a brief design doc specifying the new metrics would help ensure that they will be fit for purpose.

As an example, I wonder how the illustrative:

{{carbide_state_age_seconds{ object_type="machine", machine_id="...", dpu_id="...", normalized_state="host_init", normalized_substate="waiting_for_discovery" }}}

will apply to a multi-DPU host. (Maybe it's fine, and this is a me problem.) In general, since we may greatly increase the cardinality of our metrics planning to make sure we'll get value for it seems appropriate.