COS Terraform module tf-cos-3.0.6 stalls before worker applications are created on Canonical Kubernetes

[MrClayPole]

Summary

Deploying the COS Terraform module from tf-cos-3.0.6 on Canonical Kubernetes installed via the k8s snap does not converge.

Terraform begins creating the COS applications, but remains waiting for the Loki, Mimir, and Tempo coordinator applications. The coordinators enter a blocked state because they have no worker relation. The corresponding worker applications are never created, so the required relations cannot be established.

This is not a MicroK8s deployment and is not a migration from COS Lite.

Environment

• Kubernetes distribution: Canonical Kubernetes installed via the k8s snap

• Juju controller: foundations-maas-v3

• Juju cloud/region: cos-k8s-cloud/default

• Juju version: 3.6.23

• COS module source:

  git::https://github.com/canonical/observability-stack//terraform/cos?ref=tf-cos-3.0.6
  

• Module risk: stable

• Object storage: S3-compatible Ceph RGW

Complete main.tf

This is the complete wrapper configuration used for the failed deployment.

The S3 endpoint, bucket names, cloud name, model name, storage directives, TLS configuration, unit counts, and module/provider settings are included unchanged.

The actual S3 access key, S3 secret key, and CA-chain value are supplied separately as Terraform input variables and are not present as literal values in this file.

terraform {
  required_version = ">= 1.5"
  required_providers {
    juju = {
      source  = "juju/juju"
      version = "= 1.4.2"
    }
  }
}

provider "juju" {}

variable "s3_endpoint" {
  description = "S3 endpoint URL for COS object storage."
  type        = string
}

variable "s3_access_key" {
  description = "S3 access key for COS object storage."
  type        = string
  sensitive   = true
}

variable "s3_secret_key" {
  description = "S3 secret key for COS object storage."
  type        = string
  sensitive   = true
}

variable "s3_tls_ca_chain" {
  description = "Base64 encoded CA chain trusted by the S3 integrator."
  type        = string
  sensitive   = true
}


module "cos" {

  source = "git::https://github.com/canonical/observability-stack//terraform/cos?ref=tf-cos-3.0.6"

  model = {
    name       = "cos"
    cloud      = { name = "cos-k8s-cloud" }
    credential = "cos-k8s-cloud"
  }

  cloud         = "self-managed"
  risk          = "stable"
  anti_affinity = true
  internal_tls  = true

  s3_endpoint   = var.s3_endpoint
  s3_access_key = var.s3_access_key
  s3_secret_key = var.s3_secret_key

  loki_bucket  = "loki"
  mimir_bucket = "mimir"
  tempo_bucket = "tempo-traces"

  s3_integrator = {
    config = {
      "tls-ca-chain" = var.s3_tls_ca_chain
    }
  }

  alertmanager = { units = 1 }
  catalogue    = { units = 1 }

  grafana = {
    units = 1
    storage_directives = {
      database = "5G"
    }
  }

  loki_coordinator  = { units = 1 }
  mimir_coordinator = { units = 1 }
  tempo_coordinator = { units = 1 }

  loki_worker = {
    backend_units = 1
    read_units    = 1
    write_units   = 1
    backend_storage_directives = {
      "loki-persisted" = "10G"
    }
    read_storage_directives = {
      "loki-persisted" = "1G"
    }
    write_storage_directives = {
      "loki-persisted" = "10G"
    }
  }

  mimir_worker = {
    backend_units = 1
    read_units    = 1
    write_units   = 1
    backend_storage_directives = {
      data            = "10G"
      "recovery-data" = "5G"
    }
    read_storage_directives = {
      data            = "1G"
      "recovery-data" = "1G"
    }
    write_storage_directives = {
      data            = "10G"
      "recovery-data" = "1G"
    }
  }

  opentelemetry_collector = {
    units = 1
    storage_directives = {
      persisted = "2G"
    }
  }

  tempo_worker = {
    compactor_units         = 1
    distributor_units       = 1
    ingester_units          = 1
    metrics_generator_units = 1
    querier_units           = 1
    query_frontend_units    = 1

    compactor_worker_storage_directives = {
      wal = "1G"
    }
    distributor_worker_storage_directives = {
      wal = "1G"
    }
    ingester_worker_storage_directives = {
      wal = "5G"
    }
    metrics_generator_worker_storage_directives = {
      wal = "1G"
    }
    querier_worker_storage_directives = {
      wal = "1G"
    }
    query_frontend_worker_storage_directives = {
      wal = "1G"
    }
  }

  traefik = {
    units = 1
    storage_directives = {
      configurations = "1G"
    }
  }

  ssc = { units = 1 }
}

Steps to reproduce

1. Use a Juju model on Canonical Kubernetes installed through the k8s snap.

2. Supply valid values for the following Terraform input variables:

   s3_endpoint
   s3_access_key
   s3_secret_key
   s3_tls_ca_chain
   

3. Run:

   terraform init -upgrade -no-color
   terraform validate -no-color
   terraform plan -parallelism=80 -no-color -out=tfplan
   terraform apply -parallelism=80 -no-color -auto-approve tfplan

Expected behaviour

Terraform should create the coordinator and worker applications, establish their Juju integrations, and converge to a healthy COS deployment.

Actual behaviour

The plan completed successfully:

Plan: 105 to add, 0 to change, 0 to destroy.

During apply, Terraform continued waiting for the coordinator resources:

module.cos.module.loki.module.loki_coordinator.juju_application.loki_coordinator: Still creating...
module.cos.module.mimir.module.mimir_coordinator.juju_application.mimir_coordinator: Still creating...
module.cos.module.tempo.module.tempo_coordinator.juju_application.tempo_coordinator: Still creating...

After approximately three minutes, Juju showed:

loki   blocked  scale 1  loki-coordinator-k8s   3.7/stable   rev 88
       [consistency] Missing any worker relation.

mimir  blocked  scale 1  mimir-coordinator-k8s  2.17/stable  rev 115
       [consistency] Missing any worker relation.

tempo  blocked  scale 1  tempo-coordinator-k8s  2.10/stable  rev 162
       [consistency] Missing any worker relation.

The following applications were active:

alertmanager
ca
catalogue
grafana
loki-s3-integrator
mimir-s3-integrator
otelcol
tempo-s3-integrator
traefik

However, no Loki, Mimir, or Tempo worker applications existed:

loki-backend
loki-read
loki-write
mimir-backend
mimir-read
mimir-write
tempo-compactor
tempo-distributor
tempo-ingester
tempo-metrics-generator
tempo-querier
tempo-query-frontend

At this point, juju status --relations showed peer relations for the coordinator applications but no coordinator/worker integrations.

Terraform apply was stopped after confirming the stalled state. Terraform state contained 29 resources.

Observed sequence

1. Terraform begins creating Loki, Mimir, and Tempo coordinators.

2. Each coordinator becomes blocked with:

   [consistency] Missing any worker relation.
   

3. Terraform remains waiting for the coordinator resources.

4. The worker applications are not created.

5. Consequently, the required coordinator/worker relations cannot be created and the deployment cannot converge.

Request

Could you please confirm:

1. Whether tf-cos-3.0.6 is supported on Canonical Kubernetes installed via the k8s snap;
2. Whether this module configuration is expected to create coordinator and worker applications in this order;
3. Whether a Terraform dependency or completion-condition issue can cause coordinators to wait for worker relations before the workers are applied; and
4. Whether a supported workaround or later module release resolves this deployment deadlock.

I can provide further Juju debug logs or Terraform logs if required.