feat(monitoring): Mimir rules ConfigMap (Prometheus-free rule delivery)

[christopherfickess]

Summary

Draft POC — the Prometheus-free rule-delivery option, for the "Grafana Alloy + Mimir instead of Prometheus" direction.

Based on feat/operator-monitoring-poc (#466) so the diff here is only the Mimir addition.

Background: on a Grafana Alloy + Mimir stack, almost everything the operator emits is already backend-neutral — the ServiceMonitor and the dedicated metrics Service are consumed by Grafana Alloy (prometheus.operator.servicemonitors), which remote-writes to Mimir, and the dashboards are Grafana-native. The one Prometheus-specific piece is alert-rule delivery: Alloy/Mimir don't consume the PrometheusRule CRD. This PR adds that seam.

Adds spec.monitoring.mimirRules:

• Ships the same per-instance, pod-scoped alert rules as prometheusRule, but as a plain ConfigMap in Prometheus rules format (groups:) for a Mimir ruler-sync (mimirtool or a sidecar) to load — instead of a PrometheusRule CRD.
• Discovery label (default mimir_rules: "1") for the ruler-sync, plus an optional tenant annotation for multi-tenant Mimir.
• Cleanup-on-disable + Owns(ConfigMap), consistent with the rest of the monitoring reconcile.

No changes to ServiceMonitor / metrics Service / client+calls metrics — this is purely additive.

Cluster prerequisites (documented in docs/examples/mattermost_monitoring_mimir.yaml): install the prometheus-operator CRDs only (no Prometheus server) so Alloy can discover the ServiceMonitor; run Alloy scraping kube-state-metrics + cAdvisor (the CPU/mem/crash-loop alerts depend on those series — metrics-server does not provide them); and run a Mimir ruler-sync that loads mimir_rules-labelled ConfigMaps.

Ticket Link

Internal SRE monitoring POC — no public ticket. Alternative rule delivery for #466; companion to #467 (dashboards).

Release Note

Add `spec.monitoring.mimirRules` to the Mattermost CRD: ships Mattermost's alerting rules as a ConfigMap (Prometheus rules format) for a Grafana Mimir ruler-sync, as a Prometheus-free alternative to the PrometheusRule CRD.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Adds ConfigMap-based alert-rule delivery for Grafana Alloy and Mimir deployments without a Prometheus server.

Changes:

• Adds the spec.monitoring.mimirRules API.
• Generates and reconciles Mimir rule ConfigMaps.
• Adds generation tests and deployment documentation.

Reviewed changes

Copilot reviewed 8 out of 9 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
apis/mattermost/v1beta1/mattermost_types.go	Defines the Mimir rules API.
apis/mattermost/v1beta1/zz_generated.deepcopy.go	Adds generated deep-copy support.
config/crd/bases/installation.mattermost.com_mattermosts.yaml	Exposes the API through the CRD.
controllers/mattermost/mattermost/controller.go	Watches owned ConfigMaps.
controllers/mattermost/mattermost/monitoring.go	Reconciles and deletes rule ConfigMaps.
docs/examples/mattermost_monitoring_mimir.yaml	Documents Alloy and Mimir configuration.
pkg/mattermost/monitoring.go	Generates ConfigMaps containing alert rules.
pkg/mattermost/monitoring_test.go	Tests rule ConfigMap generation.
pkg/resources/create_resources.go	Adds ConfigMap creation support.

Files not reviewed (1)

• apis/mattermost/v1beta1/zz_generated.deepcopy.go: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[christopherfickess]

Thanks for the review — here's the resolution.

Fixed:

• Owned-only cleanup — disabling mimirRules now deletes only ConfigMaps carrying the operator's marker label (mattermost.com/mimir-rules, listed via selector), so an unrelated ConfigMap that merely shares the derived name is never touched.
• MattermostLabels map mutation — now copies spec.resourceLabels (shared with the base branch).
• Service label scoping — the Mimir rules ConfigMap uses the same corrected <name>-metrics service scoping as the PrometheusRule.

discoveryLabel vs marker-label collision: the marker label key (mattermost.com/grafana-dashboard-style mattermost.com/mimir-rules) is namespaced under mattermost.com/, so a user-supplied discoveryLabel colliding with it is not a realistic configuration; the marker is set after the discovery label and cleanup keys off the marker.

This PR is stacked on #466 (rebased so the diff is only the Mimir delta). Shared CI fixes (openapi, gofmt, CEL removal, server-side apply) come from the base.

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• apis/mattermost/v1beta1/zz_generated.deepcopy.go: Generated file

Suppressed comments (1)

controllers/mattermost/mattermost/monitoring_reconcile_test.go:40

• This reconcile test never enables MimirRules or asserts that its ConfigMap is created, remains stable, and is removed on disable. The generator test does not exercise the newly added controller/create/update/label-selected cleanup paths, so the PR's central reconciliation behavior is currently uncovered. Extend this test with the Mimir flag and ConfigMap create/delete assertions.

			Monitoring: &mmv1beta.Monitoring{
				ServiceMonitor: &mmv1beta.ServiceMonitor{Enabled: true},
				PrometheusRule: &mmv1beta.PrometheusRule{Enabled: true},

[Copilot]

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Files not reviewed (1)

• apis/mattermost/v1beta1/zz_generated.deepcopy.go: Generated file

Suppressed comments (1)

controllers/mattermost/mattermost/monitoring_reconcile_test.go:41

• This reconcile test never enables MimirRules, so the new controller create/idempotency/cleanup path is not exercised; only the pre-existing ServiceMonitor and PrometheusRule paths are covered. Add the Mimir setting plus assertions that mm-mimir-rules is created, remains stable on the second reconcile, and is deleted after disabling it.

			Monitoring: &mmv1beta.Monitoring{
				ServiceMonitor: &mmv1beta.ServiceMonitor{Enabled: true},
				PrometheusRule: &mmv1beta.PrometheusRule{Enabled: true},
			},

[Copilot]

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated 1 comment.

Files not reviewed (1)

• apis/mattermost/v1beta1/zz_generated.deepcopy.go: Generated file

Suppressed comments (1)

controllers/mattermost/mattermost/monitoring.go:46

• The existing monitoring reconcile test covers create, idempotent reconcile, and cleanup for every other monitoring resource, but it never enables or asserts the new Mimir ConfigMap path. Add the Mimir option and ConfigMap assertions so the ownership guard and label-based cleanup-on-disable are exercised; otherwise regressions in the lifecycle promised by this feature are untested.

	if err := r.checkMattermostPrometheusRule(mattermost, reqLogger); err != nil {
		return err
	}
	return r.checkMattermostMimirRules(mattermost, reqLogger)