The shared metrics-endpoint handler can fail a scrape in a few distinct ways -- the encoder queue is full (503), the encoder thread is gone (500), the Prometheus encode errors (500), the encoder panicked (500), or the reply is dropped (500). Today each is logged. CodeRabbit (on #3535) suggested also counting them.
If done
Emit a framework #[derive(Event)] counter, e.g. carbide_metrics_scrape_failures_total, labeled by a bounded outcome enum for the failure kinds above, at each failure branch of the /metrics handler -- alongside the existing logs. This is meta-observability (the endpoint counting its own partial failures); it's scrapeable on the next successful scrape, so it surfaces a busy/erroring encoder better than logs alone (aggregatable, alertable).
Deferred out of #3535 (a perf-focused PR: move the encode off the async worker), where adding a new metric family + catalogue row would be scope creep. The failures are already logged there.
The shared
metrics-endpointhandler can fail a scrape in a few distinct ways -- the encoder queue is full (503), the encoder thread is gone (500), the Prometheus encode errors (500), the encoder panicked (500), or the reply is dropped (500). Today each is logged. CodeRabbit (on #3535) suggested also counting them.If done
Emit a framework
#[derive(Event)]counter, e.g.carbide_metrics_scrape_failures_total, labeled by a bounded outcome enum for the failure kinds above, at each failure branch of the/metricshandler -- alongside the existing logs. This is meta-observability (the endpoint counting its own partial failures); it's scrapeable on the next successful scrape, so it surfaces a busy/erroring encoder better than logs alone (aggregatable, alertable).Deferred out of #3535 (a perf-focused PR: move the encode off the async worker), where adding a new metric family + catalogue row would be scope creep. The failures are already logged there.