Prometheus metrics - #110
Conversation
1cb4aca to
e4ce4bb
Compare
This introduces a first batch of simple metrics about Portail (client or server) with a TCP unprotected endpoint on port 10992 by default which respects the Prometheus protocol. How to test? ```console $ curl localhost:10992/metrics ``` UDS was attempted first but prometheus/prometheus#12024 is a thing.
|
|
||
| ACTIVE_CONNECTIONS | ||
| .with_label_values(&["ingress", proto.as_label()]) | ||
| .dec(); |
There was a problem hiding this comment.
Wouldn't that double decrement with the _guard?
| /// finished yet. | ||
| static ref ACTIVE_CONNECTIONS: IntGaugeVec = register_int_gauge_vec!( | ||
| "portail_active_connections", | ||
| "Number of active connections", |
There was a problem hiding this comment.
That might be slightly misleading for people that will use portail as a blackbox, since it also include a good part of the setup of the connection: TLS handshake, ACL eval, try backend,...
There was a problem hiding this comment.
If you have a TLS handshake with me, you have an active connection. It consumes resources.
I see your point but I think granular tracking could happen in second step.
Would you have a suggestion for a wording change on the meaning of active?
| } | ||
| }; | ||
|
|
||
| ACL_EVALUATION_TIMES |
There was a problem hiding this comment.
There are 2 code paths that lead to this metric:
- a
startbegan before the route evaluation. This seems a bit wider than just ACL evaluation, since it also include some parsing, various checks, potentially some locking. - a
startjust before looping through the backends, which will not include route evaluation.
Maybe we should split into 2 histograms even if we lose correlation?
| static ref ACL_EVALUATION_TIMES: HistogramVec = register_histogram_vec!( | ||
| "portail_acl_evaluation_seconds", | ||
| "Duration in seconds of ACL evaluations", | ||
| &["decision"] |
There was a problem hiding this comment.
This histogram receives durations in micros, I think the default buckets are not adapted https://docs.rs/prometheus/latest/src/prometheus/histogram.rs.html#25-27
Since we have a wide range from 1us to potentially 1s (or maybe more in extreme cases), maybe we should add buckets for micros on top of the default ones, like
| &["decision"] | |
| &["decision"], | |
| vec![ | |
| 0.000_001, | |
| 0.000_005, | |
| 0.000_01, | |
| 0.000_05, | |
| 0.000_1, | |
| 0.000_5, | |
| 0.001, | |
| 0.005, | |
| 0.01, | |
| 0.025, | |
| 0.05, | |
| 0.1, | |
| 0.25, | |
| 0.5, | |
| 1.0, | |
| 2.5, | |
| 5.0, | |
| 10.0, | |
| ], |
2 participants
This adds a set of simple yet interesting time series via the Prometheus metrics standard over
a UNIX domain socket (which many collectors know how to deal with)TCP socket because Prometheus cannot deal with UDS.TODO:
update nixos modulefocus on a first set of metricsFixes #11.