Skip to content

fix(redisdistributed): scope Redis alert rules to release namespace - #298

Open
PiyushSingh-ZS wants to merge 1 commit into
mainfrom
fix/issue-297
Open

fix(redisdistributed): scope Redis alert rules to release namespace#298
PiyushSingh-ZS wants to merge 1 commit into
mainfrom
fix/issue-297

Conversation

@PiyushSingh-ZS

Copy link
Copy Markdown
Collaborator

Description

charts/redisdistributed/templates/alerts.yaml rendered a per-release PrometheusRule, but 6 of 7 alert expressions had no namespace selector — they evaluated redis_* metrics across every namespace in the cluster while stamping this release's namespace/service as static labels. An alert about another namespace's Redis was mislabeled as this release.

Two were outright broken in multi-tenant clusters:

  • RedisTooManyMasterscount(redis_instance_info{role="master"}) > 1 counts every master in every namespace; with ≥2 scraped Redis instances it was permanently true (constant false positive).
  • RedisMissingMaster — only dropped to 0 if every Redis master cluster-wide disappeared, so it never fired when a single release's master was actually gone (silent failure of a critical alert).

Fix

Added namespace="{{ .Release.Namespace }}" to every metric selector, matching what RedisDown already does:

Alert Metric now scoped
RedisMissingMaster redis_instance_info
RedisTooManyMasters redis_instance_info
RedisDisconnectedSlaves redis_connected_slaves (both count/sum)
RedisReplicationBroken redis_connected_slaves
RedisClusterFlapping redis_connected_slaves
RedisRejectedConnections redis_rejected_connections_total

RedisDown was already correctly scoped and is unchanged.

Chart bumped 0.0.9 → v0.0.10, repackaged into docs/, and index.yaml regenerated.

Type of Change

  • Bug fix
  • New feature
  • Documentation update
  • Chart configuration update

Checklist

  • I have performed a self-review of my code
  • helm lint passes without errors
  • helm template renders valid YAML with all exprs namespace-scoped
  • My changes generate no new warnings
  • Chart repackaged and repo index updated

Fixes #297

The 6 alert exprs (RedisMissingMaster, RedisTooManyMasters,
RedisDisconnectedSlaves, RedisReplicationBroken, RedisClusterFlapping,
RedisRejectedConnections) evaluated redis_* metrics cluster-wide,
mislabeling other namespaces' Redis under this release. In multi-tenant
clusters RedisTooManyMasters fired permanently and RedisMissingMaster
never fired.

Add namespace="{{ .Release.Namespace }}" to every metric selector,
matching RedisDown. Bump chart to v0.0.10 and repackage.

Fixes #297
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant