diff --git a/docs/how-to/deploy-and-manage/deploy-mimir-on-juju.md b/docs/how-to/deploy-and-manage/deploy-mimir-on-juju.md new file mode 100644 index 00000000..c920ff79 --- /dev/null +++ b/docs/how-to/deploy-and-manage/deploy-mimir-on-juju.md @@ -0,0 +1,229 @@ +--- +myst: + html_meta: + description: "Deploy Grafana Mimir on Juju with a small worker topology, S3-backed storage, and a minimal remote_write validation path, without the full COS bundle." +--- + +# How to deploy Mimir on Juju + +This guide deploys Grafana Mimir on Kubernetes using +[Juju](https://documentation.ubuntu.com/juju/3.6/), Canonical's operator +lifecycle manager, without pulling in the full Canonical Observability Stack +(COS) bundle. It targets readers who already know upstream Mimir and want the +Juju-native path to a working, Mimir-focused deployment. + +If you are new to Juju, skim the [Juju documentation](https://documentation.ubuntu.com/juju/3.6/) +first — especially [Juju: get started](https://documentation.ubuntu.com/juju/3.6/tutorial/) +and the [Juju reference](https://documentation.ubuntu.com/juju/3.6/reference/) +for the meaning of terms like *model*, *application*, *unit*, and *relation*. +The runtime you get from this guide is upstream Mimir; the *charms* just handle +configuration, storage wiring, ingress, and inter-component relations. + +For a full COS deployment (Grafana, Alertmanager, Loki, etc.) instead of a +Mimir-focused one, see +[Getting started with COS on Canonical K8s](/tutorial/cos-canonical-k8s-sandbox). + +```{important} +This guide uses the Juju CLI because it is the clearest way to see how the +deployment fits together. For repeatable environments and production-oriented +rollouts, prefer the +[Mimir Terraform module](https://github.com/canonical/mimir-operators/tree/main/terraform). +``` + +## What this guide deploys + +Mimir infrastructure (required for Mimir to run): + +- `mimir-coordinator-k8s` — the public API and coordinator for the Mimir + cluster. +- `mimir-worker-k8s` — a single worker application configured to run all + Mimir roles (`role-all=true`). In production, split this into dedicated + `mimir-write`, `mimir-read`, and `mimir-backend` applications and scale each + role independently. +- `s3-integrator` (deployed as `mimir-s3`) — supplies Mimir with an S3 + endpoint and credentials. Mimir requires S3-compatible object storage; the + charm does not provide the object store itself, only the connection + details. + +Supporting infrastructure (not part of Mimir itself, but used here for a +usable end-to-end flow): + +- `traefik-k8s` — gives Mimir a stable URL for ingestion and querying from + outside the cluster. +- `opentelemetry-collector-k8s` (`otelcol`) — used only in the validation + step to scrape a test workload and push metrics to Mimir over + `prometheus_remote_write`. +- `avalanche-k8s` — a synthetic metrics generator used only in the + validation step. + +The worker also uses `role-query-frontend=true` so queries you run later go +through the same frontend path as a real deployment. + +## Prerequisites + +- Juju 3.6 or later, with a Kubernetes cloud added and a controller + bootstrapped. See + [Juju: install Juju](https://documentation.ubuntu.com/juju/3.6/howto/manage-juju/#install-juju), + [Juju: add a Kubernetes cloud](https://documentation.ubuntu.com/juju/3.6/howto/manage-clouds/#add-a-kubernetes-cloud), + and [Juju: bootstrap a controller](https://documentation.ubuntu.com/juju/3.6/howto/manage-controllers/). +- A Kubernetes cluster ready to host the Mimir model. For a local option, + see [Canonical Kubernetes: get started](https://documentation.ubuntu.com/canonical-kubernetes/latest/snap/tutorial/getting-started/). +- An S3-compatible object store that already exists, with an endpoint, + bucket, access key, and secret key. If you do not have one, see + [How to deploy Minio and S3 Integrator](../integrate/deploy-s3-integrator-and-minio) + for a disposable test store. For the connection steps in more detail, see + [How to connect object storage to Mimir on Juju](../integrate/configure-object-storage-for-mimir). +- A routable ingress address for Traefik if you want to query Mimir from + outside the cluster. See the [networking best practices](/reference/networking). + +## 1. Create a model + +A Juju [*model*](https://documentation.ubuntu.com/juju/3.6/reference/model/) +is a workspace on the controller that holds a set of related applications. +Create one dedicated to Mimir: + +```bash +juju add-model mimir +juju switch mimir +``` + +## 2. Deploy Mimir and its supporting charms + +Deploy the Mimir coordinator, one worker application, an S3 integrator, and +Traefik: + +```bash +juju deploy mimir-coordinator-k8s mimir --trust + +juju deploy mimir-worker-k8s mimir-worker \ + --trust \ + --config role-all=true \ + --config role-query-frontend=true + +juju deploy s3-integrator mimir-s3 --channel latest/stable --trust +juju deploy traefik-k8s traefik --trust +``` + +`--trust` grants the charm access to the Kubernetes API on the host cluster, +which the Mimir, S3 integrator, and Traefik charms need to create the +resources they manage. + +## 3. Point the S3 integrator at your object storage + +The `mimir-s3` application starts in `blocked` status until it has both +credentials and a target bucket. Create a Juju secret with the S3 +credentials, grant that secret to `mimir-s3`, then configure the endpoint +and bucket. + +Replace the placeholders with your existing S3-compatible storage details: + +```bash +juju add-secret mimir-s3-credentials \ + access-key= \ + secret-key= + +juju grant-secret mimir-s3-credentials mimir-s3 + +juju config mimir-s3 \ + credentials=mimir-s3-credentials \ + endpoint= \ + bucket= +``` + +If your object store needs additional settings such as a custom CA chain, +region, or path-style addressing, see +[How to connect object storage to Mimir on Juju](../integrate/configure-object-storage-for-mimir). + +## 4. Integrate the deployment + +Connect storage first, then join the worker and ingress relations: + +```bash +juju integrate mimir:s3 mimir-s3:s3-credentials +juju integrate mimir:mimir-cluster mimir-worker:mimir-cluster +juju integrate traefik:ingress mimir:ingress +``` + +Watch the model until every application reaches `active/idle`. That is the +Juju status meaning the charm has finished reconciling and is running as +configured; other transient statuses (`waiting`, `maintenance`, `blocked`) +indicate the deployment is still converging or missing something: + +```bash +juju status --relations --watch=5s +``` + +At this point Mimir should expose at least: + +- `receive-remote-write` for metrics ingestion +- `ingress` for external access +- `self-metrics-endpoint` for scrape-based validation flows + +## 5. Validate metric ingestion with OpenTelemetry Collector + +For a minimal smoke test, deploy: + +- `avalanche-k8s` — a small synthetic metrics generator. +- `opentelemetry-collector-k8s` — a scraper and forwarder charm that + collects Avalanche's metrics and pushes them to Mimir over + `prometheus_remote_write`. + +Neither of these is required for Mimir itself. They only exist here to prove +the ingestion path works end-to-end: + +```bash +juju deploy opentelemetry-collector-k8s otelcol --trust +juju deploy avalanche-k8s avalanche --trust + +juju integrate avalanche otelcol:metrics-endpoint +juju integrate otelcol:send-remote-write mimir:receive-remote-write +``` + +Wait for the applications to reach `active/idle`: + +```bash +juju status --relations --watch=5s +``` + +## 6. Query Mimir + +Use the Traefik action to get the Mimir URL: + +```bash +juju run traefik/0 show-proxied-endpoints --format=yaml \ + | yq '."traefik/0".results."proxied-endpoints"' \ + | jq +``` + +The output includes a `mimir` entry similar to: + +```json +{ + "mimir": { + "url": "http://10.43.8.34:80/mimir" + } +} +``` + +Save that URL and query Mimir's Prometheus-compatible API: + +```bash +MIMIR_URL=http://10.43.8.34:80/mimir + +curl -sG "$MIMIR_URL/prometheus/api/v1/query" \ + --data-urlencode 'query=count({__name__=~"avalanche_metric_.+"})' \ + | jq +``` + +You should see a successful query response with a value greater than `0`. + +## Next steps + +- For storage details and production-oriented caveats, see + [How to connect object storage to Mimir on Juju](../integrate/configure-object-storage-for-mimir). +- For more ways to send metrics into Mimir, including cross-model relations + and direct `remote_write` clients, see + [How to send metrics to Mimir on Juju](../integrate/send-metrics-to-mimir). +- To grow this into a full Canonical Observability Stack with Grafana, Loki, + Alertmanager, and correlated telemetry, see + [Getting started with COS on Canonical K8s](/tutorial/cos-canonical-k8s-sandbox). diff --git a/docs/how-to/deploy-and-manage/index.md b/docs/how-to/deploy-and-manage/index.md index 04d7166c..9e961558 100644 --- a/docs/how-to/deploy-and-manage/index.md +++ b/docs/how-to/deploy-and-manage/index.md @@ -21,6 +21,7 @@ Install Configure strict reproducibility Configure the Juju model Configure the Grafana database +Deploy Mimir on Juju ``` ## Secure access diff --git a/docs/how-to/deploy-and-manage/install.md b/docs/how-to/deploy-and-manage/install.md index a83ab5cd..3d970715 100644 --- a/docs/how-to/deploy-and-manage/install.md +++ b/docs/how-to/deploy-and-manage/install.md @@ -97,3 +97,10 @@ where `.n` in `tf-cos-3.0.n` is the latest available patch version in the [COS t ## Deploy COS Alerter COS Alerter is a watchdog service for COS. Deploy it on dedicated infrastructure that is separate from your COS or COS Lite deployment. For more information, including deployment details, see the [COS Alerter repository](https://github.com/canonical/cos-alerter). + +## Deploy individual components + +If you are not deploying the full Canonical Observability Stack and only need a +specific COS component on Juju, see the component-focused guides: + +- [How to deploy Mimir on Juju](deploy-mimir-on-juju) diff --git a/docs/how-to/integrate/configure-object-storage-for-mimir.md b/docs/how-to/integrate/configure-object-storage-for-mimir.md new file mode 100644 index 00000000..99a7e5f6 --- /dev/null +++ b/docs/how-to/integrate/configure-object-storage-for-mimir.md @@ -0,0 +1,125 @@ +--- +myst: + html_meta: + description: "Connect S3-compatible object storage to a Mimir on Juju deployment through s3-integrator, with production planning notes and Minio-based testing links." +--- + +# How to connect object storage to Mimir on Juju + +Mimir on Juju needs an S3-compatible object store for durable storage. The +Juju deployment does **not** create that object store for you; it expects you +to bring an existing one and connect it through the `s3-integrator` charm. + +Use this guide to connect an existing object store to a Mimir on Juju +deployment. + +## Before you begin + +You should already have: + +- A Mimir on Juju deployment (at minimum the `mimir-coordinator-k8s` + application). If you do not have one yet, follow + [How to deploy Mimir on Juju](/how-to/deploy-and-manage/deploy-mimir-on-juju) first. +- Access to an S3-compatible object store, with: + - endpoint + - bucket name + - access key + - secret key + - optional region + - optional custom CA chain for HTTPS + +If you still need a local test object store, see +[How to deploy Minio and S3 Integrator](deploy-s3-integrator-and-minio). + +## Plan your object storage + +Do this **before** wiring anything into Mimir; changing the answer to any of +these later is disruptive. + +- **Use a dedicated bucket.** Keep Mimir in its own bucket unless you have a + clear reason to share one. +- **Use dedicated credentials** for that bucket, scoped so they can only + access what Mimir needs. +- **Prefer HTTPS** for the S3 endpoint in shared or production environments, + and know in advance whether the endpoint uses a private CA (you will need + to supply the CA chain during configuration). +- **Do not use Minio in production.** The Minio charm is only for local or + disposable test environments; production planning is covered in + [Storage best practices](/reference/storage). +- **Plan capacity** around your retention and ingestion rates, and avoid + local-only or node-bound storage for production deployments. Sizing details + are in [Storage best practices](/reference/storage). +- **Keep the S3 endpoint stable** before you start wiring producers to + `receive-remote-write`; changing it later forces a reconfiguration of + Mimir workers. + +```{warning} +The Minio charm is useful for testing, but not as a production storage +strategy. For production planning, review +[Storage best practices](/reference/storage). +``` + +## 1. Deploy the S3 integrator + +```bash +juju deploy s3-integrator mimir-s3 --channel latest/stable --trust +``` + +The application will remain blocked until you give it credentials and a +target bucket. + +## 2. Create and grant the S3 credentials secret + +Create a Juju secret holding the access key and secret key, then grant that +secret to the `mimir-s3` application: + +```bash +juju add-secret mimir-s3-credentials \ + access-key= \ + secret-key= + +juju grant-secret secret: mimir-s3 +``` + +## 3. Configure the object-store location + +Set the endpoint and bucket: + +```bash +juju config mimir-s3 \ + credentials=secret: \ + endpoint= \ + bucket= +``` + +Depending on your object-store implementation, you may also need additional +options such as: + +- `region=` +- `path=` +- `s3-uri-style=path` + +If the endpoint uses a private CA, provide it as base64: + +```bash +juju config mimir-s3 tls-ca-chain="$(base64 -w0 ca-chain.pem)" +``` + +## 4. Integrate Mimir with the S3 integrator + +```bash +juju integrate mimir:s3 mimir-s3:s3-credentials +``` + +If you are still building the deployment, integrate storage **before** +joining Mimir workers to the coordinator. That keeps the storage +configuration in place before the workers read their cluster settings. + +## 5. Verify the relation + +```bash +juju status --relations +``` + +You should see an `s3` integration between `mimir` and `mimir-s3`, and both +applications should reach `active/idle`. diff --git a/docs/how-to/integrate/index.md b/docs/how-to/integrate/index.md index f351de31..9442bca4 100644 --- a/docs/how-to/integrate/index.md +++ b/docs/how-to/integrate/index.md @@ -18,6 +18,7 @@ Monitor SSL certificates with blackbox exporter Use the Loki HTTP API Use Catalogue +Send metrics to Mimir on Juju ``` ## Extend the pipeline @@ -41,10 +42,11 @@ Sync alert rules from a git repo ## Test & validate integrations -Verify your object-storage integration with a local Minio instance. +Validate local object-storage integrations or wire existing storage into Mimir. ```{toctree} :maxdepth: 1 Testing with Minio +Connect object storage to Mimir on Juju ``` diff --git a/docs/how-to/integrate/send-metrics-to-mimir.md b/docs/how-to/integrate/send-metrics-to-mimir.md new file mode 100644 index 00000000..d6bcec25 --- /dev/null +++ b/docs/how-to/integrate/send-metrics-to-mimir.md @@ -0,0 +1,183 @@ +--- +myst: + html_meta: + description: "Send metrics from applications outside Juju into a Mimir on Juju deployment by using the OpenTelemetry Collector snap as a scraper and remote_write client." +--- + +# How to send metrics to Mimir on Juju + +Use this guide to send metrics from an application that is **not** managed by +Juju into a Mimir on Juju deployment. + +## What this guide sets up + +Mimir on Juju accepts metrics over Prometheus `remote_write` through its +`receive-remote-write` relation. When the sender is another charm, that +relation does everything for you. When the sender is an uncharmed workload +(for example, a plain systemd service on a VM), there is no relation to +attach to, so you need something outside Juju that can: + +1. scrape or receive metrics from the workload, and +2. push them into Mimir's `remote_write` endpoint through the Mimir + ingress URL. + +We use the [OpenTelemetry Collector +snap](https://snapcraft.io/opentelemetry-collector) for this. The **snap** +(not the charm) is a standalone binary you install directly on the machine +that runs, or can reach, the uncharmed workload. It is configured through a +YAML file placed in `/etc/otelcol/config.d/`. We recommend running it as +close as possible to the workload to minimize network hops and reduce the +chance of losing telemetry during transient failures. + +The end result is: `uncharmed workload` -> `otelcol snap` -(remote_write over +Traefik ingress)-> `Mimir on Juju`. + +For a general primer on this pattern with the full COS Lite stack, see +[How to integrate COS Lite with uncharmed applications](integrating-cos-lite-with-uncharmed-applications). +For the Mimir deployment itself, see +[How to deploy Mimir on Juju](/how-to/deploy-and-manage/deploy-mimir-on-juju). + +## Prerequisites + +- A working Mimir on Juju deployment with ingress enabled (see + [How to deploy Mimir on Juju](/how-to/deploy-and-manage/deploy-mimir-on-juju)). +- An uncharmed workload that exposes Prometheus-format metrics. +- A machine where you can install the OpenTelemetry Collector snap and that + can: + - reach the workload metrics endpoint + - reach the Mimir ingress URL + +## 1. Get the Mimir URL + +In the Juju model where Mimir is deployed, run: + +```bash +juju run traefik/0 show-proxied-endpoints --format=yaml \ + | yq '."traefik/0".results."proxied-endpoints"' \ + | jq +``` + +Look for the `mimir` entry: + +```json +{ + "mimir": { + "url": "http://10.43.8.34:80/mimir" + } +} +``` + +You will use: + +- `/api/v1/push` for ingestion +- `/prometheus/api/v1/query` for verification + +## 2. Install the OpenTelemetry Collector snap + +On the machine that can reach the uncharmed workload: + +```bash +sudo snap install opentelemetry-collector +``` + +## 3. Create the collector configuration + +Write a collector config file under `/etc/otelcol/config.d/`. The snap loads +every YAML file it finds in that directory and merges them, so you can drop +in additional files later for more workloads without editing the existing +one. + +The pipeline below scrapes a workload that exposes metrics on +`http://my-workload:9000/metrics` and forwards them to Mimir over +`remote_write`. The `prometheus` receiver acts as the scraper, and the +`prometheusremotewrite` exporter pushes into Mimir's ingestion URL. The +`X-Scope-OrgID` header is required by Mimir's multi-tenant API; use +`anonymous` for a single-tenant deployment. + +```yaml +receivers: + prometheus: + config: + scrape_configs: + - job_name: "my-workload" + scrape_interval: 15s + static_configs: + - targets: ["my-workload:9000"] + +exporters: + prometheusremotewrite: + endpoint: "http://10.43.8.34:80/mimir/api/v1/push" + headers: + X-Scope-OrgID: anonymous + +service: + pipelines: + metrics: + receivers: [prometheus] + exporters: [prometheusremotewrite] +``` + +Save it as: + +```bash +sudo mkdir -p /etc/otelcol/config.d +sudo editor /etc/otelcol/config.d/otelcol_mimir.yaml +``` + +Replace: + +- `my-workload:9000` with the address of your workload's metrics endpoint +- `http://10.43.8.34:80/mimir/api/v1/push` with your actual Mimir ingress URL + +## 4. Restart the collector + +```bash +sudo snap restart opentelemetry-collector +``` + +If you want to confirm the collector started cleanly: + +```bash +sudo snap logs opentelemetry-collector +``` + +## 5. Verify that metrics reached Mimir + +Query Mimir's Prometheus-compatible API: + +```bash +MIMIR_URL=http://10.43.8.34:80/mimir + +curl -sG "$MIMIR_URL/prometheus/api/v1/query" \ + --data-urlencode 'query=up{job="my-workload"}' \ + | jq +``` + +If the pipeline is working, the query returns a successful result for the +`my-workload` job. + +## HTTPS and private CAs + +If either the workload metrics endpoint or the Mimir ingress URL uses TLS +signed by a private CA, install that CA into the machine trust store used by +the snap. + +See the CA-handling guidance in +[How to integrate COS Lite with uncharmed applications](integrating-cos-lite-with-uncharmed-applications). + +## Adapting the scrape config + +The example above uses a single static target, but the same pattern works +for: + +- multiple targets in one job +- multiple scrape jobs +- custom `metrics_path` values +- authenticated or HTTPS endpoints + +The important part is that the metrics pipeline ends with the +`prometheusremotewrite` exporter pointing at: + +```text +/api/v1/push +``` diff --git a/docs/redirects.txt b/docs/redirects.txt index 7b9cc9eb..e5a2b5fb 100644 --- a/docs/redirects.txt +++ b/docs/redirects.txt @@ -12,3 +12,4 @@ "design-goals" "explanation/architecture/design-goals" "juju-topology" "explanation/architecture/juju-topology" "security" "reference/security-hardening-guide" +"tutorial/mimir-on-juju" "how-to/deploy-and-manage/deploy-mimir-on-juju" diff --git a/docs/tutorial/index.md b/docs/tutorial/index.md index 78af36c7..e0bf314d 100644 --- a/docs/tutorial/index.md +++ b/docs/tutorial/index.md @@ -36,4 +36,3 @@ COS on Canonical K8s COS Lite on Canonical K8s COS Lite on MicroK8s ``` -