Traefik route conflicts between otel and tempo in the COS stack

[adhityaravi]

In the COS stack, both tempo and otelcol request the same listener ports / entry points from traefik. And they request trafik that traffic from these ports to be routed to their own backend service. For example look at the entry points in traefiks static configuration

entryPoints:
  diagnostics:
    address: :8082
  health:
    address: :13133
  jaeger-grpc:
    address: :14250
  jaeger-thrift-http:
    address: :14268
  loki-http:
    address: :3500
  metrics:
    address: :8888
  otlp-grpc:
    address: :4317
  otlp-http:
    address: :4318
  tempo-grpc:
    address: :9096
  tempo-http:
    address: :3200
  web:
    address: :80
  websecure:
    address: :443
  zipkin:
    address: :9411
global:
  checknewversion: false
log:
  level: DEBUG
metrics:
  prometheus:
    addRoutersLabels: true
    addServicesLabels: true
    entryPoint: diagnostics
ping:
  entryPoint: diagnostics
providers:
  file:
    directory: /opt/traefik/juju
    watch: true

The following entry points are common between both tempo and otelcol

  jaeger-grpc:
    address: :14250
  jaeger-thrift-http:
    address: :14268
  otlp-grpc:
    address: :4317
  otlp-http:
    address: :4318
  zipkin:
    address: :9411

and the look at the dynamic configs submitted by both the apps. otelcol first:

http:
  routers:
    juju-cos-dev-otelcol-health:
      entryPoints:
      - health
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-health
    juju-cos-dev-otelcol-health-tls:
      entryPoints:
      - health
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-health
      tls: {}
    juju-cos-dev-otelcol-jaeger-grpc:
      entryPoints:
      - jaeger-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-jaeger-grpc
    juju-cos-dev-otelcol-jaeger-grpc-tls:
      entryPoints:
      - jaeger-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-jaeger-grpc
      tls: {}
    juju-cos-dev-otelcol-jaeger-thrift-http:
      entryPoints:
      - jaeger-thrift-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-jaeger-thrift-http
    juju-cos-dev-otelcol-jaeger-thrift-http-tls:
      entryPoints:
      - jaeger-thrift-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-jaeger-thrift-http
      tls: {}
    juju-cos-dev-otelcol-loki-http:
      entryPoints:
      - loki-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-loki-http
    juju-cos-dev-otelcol-loki-http-tls:
      entryPoints:
      - loki-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-loki-http
      tls: {}
    juju-cos-dev-otelcol-metrics:
      entryPoints:
      - metrics
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-metrics
    juju-cos-dev-otelcol-metrics-tls:
      entryPoints:
      - metrics
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-metrics
      tls: {}
    juju-cos-dev-otelcol-otlp-grpc:
      entryPoints:
      - otlp-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-otlp-grpc
    juju-cos-dev-otelcol-otlp-grpc-tls:
      entryPoints:
      - otlp-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-otlp-grpc
      tls: {}
    juju-cos-dev-otelcol-otlp-http:
      entryPoints:
      - otlp-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-otlp-http
    juju-cos-dev-otelcol-otlp-http-tls:
      entryPoints:
      - otlp-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-otlp-http
      tls: {}
    juju-cos-dev-otelcol-zipkin:
      entryPoints:
      - zipkin
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-zipkin
    juju-cos-dev-otelcol-zipkin-tls:
      entryPoints:
      - zipkin
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-otelcol-service-zipkin
      tls: {}
  services:
    juju-cos-dev-otelcol-service-health:
      loadBalancer:
        servers:
        - url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:13133
    juju-cos-dev-otelcol-service-jaeger-grpc:
      loadBalancer:
        servers:
        - url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:14250
    juju-cos-dev-otelcol-service-jaeger-thrift-http:
      loadBalancer:
        servers:
        - url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:14268
    juju-cos-dev-otelcol-service-loki-http:
      loadBalancer:
        servers:
        - url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:3500
    juju-cos-dev-otelcol-service-metrics:
      loadBalancer:
        servers:
        - url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:8888
    juju-cos-dev-otelcol-service-otlp-grpc:
      loadBalancer:
        servers:
        - url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:4317
    juju-cos-dev-otelcol-service-otlp-http:
      loadBalancer:
        servers:
        - url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:4318
    juju-cos-dev-otelcol-service-zipkin:
      loadBalancer:
        servers:
        - url: http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:9411

This says, for example, the zipkin entrypoint should be forwarded to the backend service juju-cos-dev-otelcol-service-zipkin ie to the address http://otelcol-0.otelcol-endpoints.cos-dev.svc.cluster.local:9411.

Lets look at the dynamic config requested by tempo now,

http:
  routers:
    juju-cos-dev-tempo-jaeger-grpc:
      entryPoints:
      - jaeger-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-jaeger-grpc
    juju-cos-dev-tempo-jaeger-grpc-tls:
      entryPoints:
      - jaeger-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-jaeger-grpc
      tls: {}
    juju-cos-dev-tempo-jaeger-thrift-http:
      entryPoints:
      - jaeger-thrift-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-jaeger-thrift-http
    juju-cos-dev-tempo-jaeger-thrift-http-tls:
      entryPoints:
      - jaeger-thrift-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-jaeger-thrift-http
      tls: {}
    juju-cos-dev-tempo-otlp-grpc:
      entryPoints:
      - otlp-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-otlp-grpc
    juju-cos-dev-tempo-otlp-grpc-tls:
      entryPoints:
      - otlp-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-otlp-grpc
      tls: {}
    juju-cos-dev-tempo-otlp-http:
      entryPoints:
      - otlp-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-otlp-http
    juju-cos-dev-tempo-otlp-http-tls:
      entryPoints:
      - otlp-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-otlp-http
      tls: {}
    juju-cos-dev-tempo-tempo-grpc:
      entryPoints:
      - tempo-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-tempo-grpc
    juju-cos-dev-tempo-tempo-grpc-tls:
      entryPoints:
      - tempo-grpc
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-tempo-grpc
      tls: {}
    juju-cos-dev-tempo-tempo-http:
      entryPoints:
      - tempo-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-tempo-http
    juju-cos-dev-tempo-tempo-http-tls:
      entryPoints:
      - tempo-http
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-tempo-http
      tls: {}
    juju-cos-dev-tempo-zipkin:
      entryPoints:
      - zipkin
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-zipkin
    juju-cos-dev-tempo-zipkin-tls:
      entryPoints:
      - zipkin
      rule: ClientIP(`0.0.0.0/0`)
      service: juju-cos-dev-tempo-service-zipkin
      tls: {}
  services:
    juju-cos-dev-tempo-service-jaeger-grpc:
      loadBalancer:
        servers:
        - url: h2c://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:14250
    juju-cos-dev-tempo-service-jaeger-thrift-http:
      loadBalancer:
        servers:
        - url: http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:14268
    juju-cos-dev-tempo-service-otlp-grpc:
      loadBalancer:
        servers:
        - url: h2c://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:4317
    juju-cos-dev-tempo-service-otlp-http:
      loadBalancer:
        servers:
        - url: http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:4318
    juju-cos-dev-tempo-service-tempo-grpc:
      loadBalancer:
        servers:
        - url: h2c://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:9096
    juju-cos-dev-tempo-service-tempo-http:
      loadBalancer:
        servers:
        - url: http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:3200
    juju-cos-dev-tempo-service-zipkin:
      loadBalancer:
        servers:
        - url: http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:9411

Now this says, the same entrypoint zipkin should be forwarded to the backend service juju-cos-dev-tempo-service-zipkin ie to the address http://tempo-0.tempo-endpoints.cos-dev.svc.cluster.local:9411.

There are 5 such conflcts in the COS stacks default deployment. So which backend is actually being used? In my tests it was tempo. how? why? I dont know because traefik does not document this behavior which is weird. But its also weird to wire the same entry point to different backends without any path prefix and expect sane results. This feels like a critical network topology hole in the architecture which would need to be addressed urgently.

In the istio-ingress-k8s charm, we handle this by detecting such conflicts and putting the ingress charm in a blocked state with an appropriate message and logs and ingress is provided to neither of the conflicting charms. But traefik silently lets the faulty configuration to cascade.

[MichaelThamm]

In the future, we will move to having only ingress for opentelemetry-collector and grafana:

• Since otelcol will be the single entrypoint for ingesting telemetry in COS.
• Grafana is the only UI ingressed in prod deployments bc it is the only one supporting Oauth.
  • It also offers a UI for all telemetry types and alerting

For this reason, I think the value for ingressed Tempo is thin. I know that the default deployment behavior is ingress everything, which causes this issue on default deployments: bad. IMO, we should:

1. remove the ability to ingress Tempo
2. relate tempo and otelcol internally via tracing relations

I know there are a few shortcomings with Otelcol currently like:

• Otelcol should publish its service FQDN instead of per pod headless service address in the relation data opentelemetry-collector-k8s-operator#272
• Receivers duplicate data when scaled, without Traefik ingress opentelemetry-collector-k8s-operator#188

which we should address sooner than later: especially if we move in the proposed direction.

[adhityaravi]

That sounds reasonable to me. But I'll let you and the COS owners decide the way forward. But if you decide to drop the ingress to tempo, I feel it should be the same for all the backend applications (mimir and loki) or it's going to be weird.

Also if otel needs to be ingressed and tempo removed from ingress, the following issue (and the issue related to it) need to be fixed as well

• Ingressed Otelcol should support gRPC

Plus currently the stack offers endpoints directly from the backend applications (tempo, mimir and loki) this should probably be changed to Otelcols ingestion endpoints.

[PietroPasotti]

And by 'remove the ability to ingress' do you mean dropping the ingress relations from the charm? or dropping the relations from our terraform plans? I'd be +1 for the second one, but we have to think about whether we want to force tempo-charm users to also deploy otelcol just so they can expose their ingestion endpoints.

[MichaelThamm]

@PietroPasotti just from the COS TF module. Internally we would integrate otelcol:send-traces tempo and externally foo otelcol:receive-otlp

[MichaelThamm]

The problem is the proposed solution will take a long time to land. I think a temporary solution is to:

1. default otelcol to ingress=false
2. add variable validation which stops the user from ingressing both otelcol and tempo. This would limit the usefulness of COS to some degree, but at least the user won't be surprised when telemetry intended for otelcol is being sent to Tempo