fix: stop Prometheus rejecting the ingest counters, cut 0.3.1 - #28
Merged
Conversation
Three faults stacked behind one HTTP 400 every five or six minutes, found by dumping the rejected payload through a debug exporter on metrics/count. - batch sat after deltatocumulative and handed it datapoints whose timestamps had regressed, ~1/min. Removing it took delta.ErrOutOfOrder from 55 per collector hour to none. - The count connector emits one ResourceMetrics per incoming batch and merges none of them, so a spoke shipping 50 log batches at once made 50 samples for one series microseconds apart. interval collapses a stream to one sample per 30s scrape. - The counters inherited the source telemetry's own timestamp, so a log backlog minted counter samples up to 11 hours old, past the 30m out_of_order_time_window. They are stamped at ingest now, which is what an ingest counter means. It has to run after interval, or several datapoints of one stream would restamp into duplicate samples. - deltatocumulative tracked an unbounded number of streams. The counters carry the sender's service.instance.id, so every service restart added one. Capped at 5000. - The counters export through their own otlp_http/prometheus_count, so send_failed_metric_points is attributable by pipeline rather than pooling counter drops with a spoke's application metrics. telemetry_logs_total now reports all eight senders, where five had never reached Prometheus at all, and none of them reset. That counter is what ProjectTelemetrySilent and ProjectsUncovered read.
`just demo` runs a throwaway stack under its own compose project and is torn down again, so the rule fires against the demo's own hub. Bootstrapping the pair instead would leave a ProjectTelemetrySilent rule firing forever once the demo is removed. - Exempt as a pair, not by project name, so a real project called demo in a real environment is still caught. - Re-rendering through bootstrap.sh also picked up a coverage.yaml that had drifted from the template's `unknown` wording.
The project and env arguments are validated, but the pairs read back from the rendered files' '# COVERS:' markers were not, and they land in a sed replacement and a PromQL label value. Fail the run instead of rendering a rule that silently never matches.
The ceiling was 67 spokes away at ~1,400 series each, and 100k would have put Prometheus near its 2g mem_limit before the warning arrived. A single static threshold also cannot separate the two things it was being asked to catch: onboarding is slow and planned, an exploding label is fast and is the failure that hurts. - PrometheusCardinalityHigh drops to 30k, ~4x the ~7k baseline and ~18 spokes of room, firing at roughly a quarter of Prometheus's memory limit. - New PrometheusCardinalitySpike on 5,000 new series in 30 minutes, which needs no re-tuning as the fleet grows. Calibrated on the hub: steady state moves under 100 series/hour, one spoke onboarding adds ~1,400. Measured over 30m so the window fits inside the shared relativeTimeRange, and a head-block truncation only ever moves the delta negative, so it cannot false-fire. - RUNBOOK quotes both numbers.
- extract valid_pair() so argv and on-disk COVERS markers share one check - drop file_storage from the count exporter's queue; the counters resume after a restart, so in-flight samples are not worth a queue on disk - note that the max_streams heap figure predates the interval processor - condense the changelog and date the release
simonvanlierde
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes the three loose ends in section 7 of the post-merge list. Cuts 0.3.1.
Ingest counters rejected every 5–6 minutes
Three faults in
metrics/count, each found by dumping the rejected payload through a temporarydebugexporter:batchafterdeltatocumulativereordered timestamps, ~1/min. Removed.interval: 30scollapses them.intervalso the restamp cannot recreate duplicates.telemetry_logs_totalnow reports 8 senders where it reported 3; the other five were rejected wholesale. Both coverage alerts read that counter. No application telemetry was lost, which the newotlp_http/prometheus_countexporter proved by separating the two failure counts.deltatocumulativeis capped at 5000 streams.Also
ProjectsUncoveredexemptsdemo/demo; bootstrapping it would leaveProjectTelemetrySilentfiring forever after teardown.bootstrap.shrefuses malformed# COVERS:markers.PrometheusCardinalityHighlowered to 30k; newPrometheusCardinalitySpikeon 5,000 series in 30m.Verification
Applied to the running hub for 16 minutes: zero rejections,
otelcol_exporter_send_failed_metric_pointsabsent, all 12 rules healthy,just checkpasses.Tag
v0.3.1after merge.