feat: COS-Dev mesh pt.2

.github/prompts/review-docs.prompt.md

@@ -13,8 +13,10 @@ Review the documentation for clarity, completeness, and accuracy.
 - H1 titles under `docs/how-to` should start with "How to".
 - Section headers and index entries should all use sentence case (not title case).
 - Known product names should be capitalized consistently throughout the documentation.
+- Spelling according to US English conventions.
 - Changes in docs should be reflected in the glossary, if the changes are related to terms/concepts that are unique to
   COS or charmed observability (don't redefine all/general terms).
+- DO NOT use prompt marks (e.g. $ or #) in code samples.
 
 ## Context
 

docs/.custom_wordlist.txt

@@ -172,6 +172,7 @@ SHA
 sitemapindex
 SLAs
 Snapcraft
+snap_microk8s
 SNMP
 Sphinx
 Spread

docs/explanation/telemetry/logging-architecture.md

@@ -89,7 +89,7 @@ relations:
 this results in an auto-render Promtail config file with three scrape jobs, one for each "filename":
 
 ```bash
-$ juju ssh --container postgresql pgsql/0 cat /etc/promtail/promtail_config.yaml
+juju ssh --container postgresql pgsql/0 cat /etc/promtail/promtail_config.yaml
 ```
 
 ```yaml
@@ -163,7 +163,7 @@ relations:
 This results in an auto-generated `/etc/otelcol/config.d/otelcol_0.yaml` config file with juju topology labels and the default scrape jobs for `/var/log/**/*log` and `journalctl`:
 
 ```bash
-$ juju ssh otelcol/0 cat /etc/otelcol/config.d/otelcol_0.yaml
+juju ssh otelcol/0 cat /etc/otelcol/config.d/otelcol_0.yaml
 ```
 
 ```yaml

docs/explanation/telemetry/telemetry-labels.md

@@ -22,9 +22,9 @@ By convention, applications expose labeled metrics under a [`/metrics` endpoint]
 For example, you can run the prometheus application and curl its `:9090/metrics` endpoint to obtain the metrics exposed by the process.
 
 ```bash
-$ sudo snap install prometheus
+sudo snap install prometheus
 
-$ curl localhost:9090/metrics
+curl localhost:9090/metrics
 
 # -- snip --
 
@@ -69,7 +69,7 @@ scrape_configs:
 Labels that are specified under a `static_configs` entry are automatically attached to all metrics scraped from the targets:
 
 ```bash
-$ curl -s --data-urlencode 'match[]={__name__="prometheus_http_requests_total"}' localhost:9090/api/v1/series | jq '.data'
+curl -s --data-urlencode 'match[]={__name__="prometheus_http_requests_total"}' localhost:9090/api/v1/series | jq '.data'
 [
   {
     "__name__": "prometheus_http_requests_total",

docs/how-to/configure-and-tune/disable-charmed-rules.md

@@ -24,7 +24,7 @@ using a boolean configuration option, called `forward_alert_rules`:
 For example, to disable forwarding of all alert rules from opentelemetry collector,
 
 ```
-$ juju config opentelemetry-collector forward_alert_rules=false
+juju config opentelemetry-collector forward_alert_rules=false
 ```
 
 ## Silence charmed rules using alertmanager configuration

docs/how-to/configure-and-tune/evaluate-telemetry-volume.md

@@ -11,12 +11,27 @@ In order to correctly size the VM(s) needed for COS, you need to know how much t
 
 
 ## Metrics rate
+
+### Manual evaluation
+Find out the metrics endpoint manifest for each observed workload. If it is not documented,
+you will need to manually count the number of non-comment lines served on the metrics endpoint,
+for example:
+
+```bash
+curl -sf localhost:8080/metrics | grep -v "^# " | wc -l
+```
+
+This will give you the number of time series that will be created for the workload, per unit.
+
+Another option is to deploy a temporary pilot Prometheus charm.
+
+### With charmed Prometheus
 Have your deployment sending all metrics to Prometheus (or Mimir) and inspect the 48hr plot for `count({__name__=~".+"})`.
 The raw data can also be obtained by querying the Prometheus `query` endpoint directly:
 
 ```bash
 curl -s --data-urlencode 'query=count({__name__=~".+"})' \
-  ${PROM_UNIT_IP}:9090/api/v1/query"
+  http://${PROM_UNIT_IP}:9090/api/v1/query
 ```
 
 Compare the output to the number of metrics exposed by individual applications.
@@ -36,8 +51,13 @@ load[load generator] ---|db| postgresql
 postgresql ---|metrics-endpoint| prometheus
 ```
 
-
 ## Logs rate
+### Manual evaluation
+The most reliable way to evaluate the logging rate of a workload is with load tests.
+
+Another option is to deploy temporary pilot Loki and Prometheus charms.
+
+### With charmed Loki and Prometheus
 Have your deployment sending all logs to Loki, and inspect the 48hr plot for `loki_distributor_*_received_total`:
 
 ```

docs/how-to/configure-and-tune/selectively-drop-telemetry-scrape-config.md

@@ -52,7 +52,7 @@ graph LR
 We can specify the `drop` action via a config option for the [scrape-config charm](https://charmhub.io/prometheus-scrape-config-k8s):
 
 ```shell
-$ juju config sc metric_relabel_configs="$(cat <<EOF
+juju config sc metric_relabel_configs="$(cat <<EOF
 - source_labels: ["__name__"]
   regex: "scrape_samples_.+"
   action: "drop"

docs/how-to/deploy-and-manage/index.md

@@ -14,14 +14,10 @@ These guides cover deploying, upgrading, managing, and securing access to COS.
 
 See our [tutorials](/tutorial/index) for guidance on deploying COS.
 
-## Upgrades
-
-Move between COS revisions with confidence.
-
 ```{toctree}
 :maxdepth: 1
 
-Cross-track upgrade instructions <upgrade>
+Install <install>
 ```
 
 ## Secure access
@@ -34,3 +30,14 @@ Protect and expose COS endpoints for production traffic.
 Configure TLS encryption <configure-tls-encryption>
 Configure ingress <configure-granular-ingress>
 ```
+
+## Upgrades
+
+Move between COS revisions with confidence.
+
+```{toctree}
+:maxdepth: 1
+
+Cross-track upgrade instructions <upgrade>
+```
+

docs/how-to/deploy-and-manage/install.md

@@ -0,0 +1,106 @@
+---
+myst:
+ html_meta:
+   description: "Install the Canonical Observability Stack: preparation checklist covering sizing, networking, storage, and deployment options."
+---
+
+# How to install COS
+
+This guide walks you through the prerequisites and deployment planning needed to install the Canonical Observability Stack (COS). Use it to prepare your environment, create a Terraform plan, and deploy supporting components in the recommended order.
+
+## Preparation
+
+Before deploying COS or COS Lite, work through the items below.
+
+### COS flavor
+
+The [flavor of COS](/explanation/overview/what-is-cos) to install depends on your use case.
+If you want to install on edge devices, want to rely on local storage, or do not need high availability, then COS Lite is likely the right choice; otherwise
+you should probably go with "full" COS.
+
+```{mermaid}
+graph LR
+
+subgraph env["Monitored environment"]
+opentelemetry-collector
+end
+
+subgraph k8s["K8s cluster"]
+COS
+end
+
+subgraph pc["Public cloud"]
+cos-alerter["COS Alerter"]
+end
+
+subgraph storage["Storage cluster"]
+S3
+end
+
+opentelemetry-collector ---|telemetry| COS
+COS --- S3
+COS --- cos-alerter
+```
+
+### Kubernetes cluster
+
+Deploy COS on a high-availability Kubernetes cluster with at least 3 control plane nodes.
+
+### Sizing
+
+Use the [sizing guide](/reference/system-requirements) to determine the minimum hardware for your deployment.
+If you don't yet know how much telemetry your workloads generate, start with [How to evaluate telemetry volume](/how-to/configure-and-tune/evaluate-telemetry-volume).
+
+Follow the [storage best practices](/reference/storage) to set up a distributed storage backend with a replication factor of 3.
+Do **not** use `hostPath` storage in production.
+
+### Configure networking
+
+Review the [networking best practices](/reference/networking) and ensure:
+
+- A load balancer (for example, MetalLB) is available to give Traefik a stable IP.
+- Egress is open for Charmhub, the Juju OCI registry, and Snapcraft.
+
+### Plan for TLS
+
+Production deployments should use TLS.
+See [How to configure TLS encryption](/how-to/deploy-and-manage/configure-tls-encryption) for the available modes and what you need to prepare (for example, an external certificates provider).
+
+### Authentication and authorization
+Only the Grafana and Traefik charms support authentication.
+To expose Grafana publicly, deploy two Traefik charms: one for internal connections and another for external access to provide ingress.
+
+### Dedicated Juju controller and model
+
+You should bootstrap a dedicated Juju controller and model just for COS.
+
+## Terraform plan
+
+Create a `main.tf` file like this one:
+
+```hcl
+resource "juju_model" "cos" {
+  name = "cos"
+}
+
+module "cos" {
+  source = "git::https://github.com/canonical/observability-stack//terraform/cos?ref=tf-cos-3.0.n"
+  risk         = "stable"
+  model_uuid   = juju_model.cos.uuid
+  s3_endpoint   = "http://IP_ADDRESS:PORT"
+  s3_secret_key = "REPLACE_ME"
+  s3_access_key = "REPLACE_ME"
+}
+```
+
+### Revision pins
+Revision pinning is optional. With revision pins, subsequent `terraform apply` invocation will not refresh charms. Without revision pins, each `terraform apply` would refresh to the latest revision in track, if a new one released.
+where `.n` in `tf-cos-3.0.n` is the latest available patch version in the [COS tags](https://github.com/canonical/observability-stack/tags) list.
+
+### Revision pins
+
+Deploying COS without revision pins, per component, will deploy the latest charms revisions in-track. Any subsequent Terraform plans will experience the same behaviour i.e., keeping COS up-to-date. However, if you require more stability, it is advised to pin the charm revisions of all components.
+
+## Deploy COS Alerter
+
+COS Alerter is a watchdog service for COS. Deploy it on dedicated infrastructure that is separate from your COS or COS Lite deployment. For more information, including deployment details, see the [COS Alerter repository](https://github.com/canonical/cos-alerter).

docs/how-to/integrate/add-tracing-to-cos-lite.md

@@ -15,7 +15,7 @@ In the same Juju model as you have COS Lite deployed, deploy the ``tempo-coordin
 using the following command: 
 
 ```bash
-$ juju deploy tempo-coordinator-k8s tempo \
+juju deploy tempo-coordinator-k8s tempo \
     --channel edge \
     --trust
 ```
@@ -28,7 +28,7 @@ with the worker nodes directly.
 ## Deploy the Tempo Worker
 
 ```bash    
-$ juju deploy tempo-worker-k8s tempo-worker \
+juju deploy tempo-worker-k8s tempo-worker \
     --channel edge \
     --trust
 ```
@@ -47,7 +47,7 @@ If you don't have an s3 bucket ready at hand, follow [this guide](https://discou
 Once you're done deploying ``minio`` and ``s3``, you can run:
 
 ```bash
-$ juju integrate tempo s3
+juju integrate tempo s3
 ```
 
 And wait for the `tempo` application to go to `active/idle`.
@@ -56,7 +56,7 @@ And wait for the `tempo` application to go to `active/idle`.
 ## Integrate coordinator and workers
 
 ```bash    
-$ juju integrate tempo tempo-worker
+juju integrate tempo tempo-worker
 ```
 
 At this point your `juju status` should look like this:
@@ -73,29 +73,29 @@ Coordinator is reporting 'degraded' because not all roles are assigned in the re
 You can enable self-monitoring for ``tempo`` by integrating it with the other COS Lite components.
 
 ```bash
-$ juju integrate loki:logging tempo:logging
-$ juju integrate s3:s3-credentials tempo:s3
-$ juju integrate tempo:grafana-dashboard grafana:grafana-dashboard
-$ juju integrate tempo:grafana-source grafana:grafana-source
-$ juju integrate tempo:metrics-endpoint prometheus:metrics-endpoint
-$ juju integrate tempo:tempo-cluster tempo-worker:tempo-cluster
-$ juju integrate traefik:traefik-route tempo:ingress
+juju integrate loki:logging tempo:logging
+juju integrate s3:s3-credentials tempo:s3
+juju integrate tempo:grafana-dashboard grafana:grafana-dashboard
+juju integrate tempo:grafana-source grafana:grafana-source
+juju integrate tempo:metrics-endpoint prometheus:metrics-endpoint
+juju integrate tempo:tempo-cluster tempo-worker:tempo-cluster
+juju integrate traefik:traefik-route tempo:ingress
 ```
 
 Similarly, you can enable tracing in COS Lite by integrating the COS Lite charms that support 
 it to `tempo` over the `tracing` relation:
 
 ```bash 
-$ juju integrate tempo:tracing alertmanager:tracing
-$ juju integrate tempo:tracing catalogue:tracing
-$ juju integrate tempo:tracing traefik:charm-tracing
-$ juju integrate tempo:tracing traefik:workload-tracing
-$ juju integrate tempo:tracing loki:charm-tracing
-$ juju integrate tempo:tracing loki:workload-tracing
-$ juju integrate tempo:tracing grafana:charm-tracing
-$ juju integrate tempo:tracing grafana:workload-tracing
-$ juju integrate tempo:tracing prometheus:charm-tracing
-$ juju integrate tempo:tracing prometheus:workload-tracing
+juju integrate tempo:tracing alertmanager:tracing
+juju integrate tempo:tracing catalogue:tracing
+juju integrate tempo:tracing traefik:charm-tracing
+juju integrate tempo:tracing traefik:workload-tracing
+juju integrate tempo:tracing loki:charm-tracing
+juju integrate tempo:tracing loki:workload-tracing
+juju integrate tempo:tracing grafana:charm-tracing
+juju integrate tempo:tracing grafana:workload-tracing
+juju integrate tempo:tracing prometheus:charm-tracing
+juju integrate tempo:tracing prometheus:workload-tracing
 ```
 
 ```{note}
@@ -107,7 +107,7 @@ You can also achieve the same by running ``jhack imatrix fill``.
 If you have a charm offering a `certificates` endpoint such as [`self-signed-certificates`](https://charmhub.io/self-signed-certificates), you can integrate it with `tempo`:
 
 ```bash
-$ juju integrate tempo:certificates ca:certificates 
+juju integrate tempo:certificates ca:certificates 
 ```
 
 to enable traces to be sent to `tempo` over HTTPS (or gRPCs).