Centralized observability for Insight instances: Loki + Grafana (logs, metrics, alerting, fleet status)

[mitasovr]

Part of #1786 (Quality: Reliability vector). This task implements its plan items: monitoring over application metrics, telemetry collection from instances/services, and a secure connection between client instances and the telemetry service. The dashboards/alerts below should expose the #1786 target metrics (uptime > 99.99%, successful connector syncs > 99.99%, failed upgrades/deployments).

Problem

Today we have almost no observability across Insight instances. Logs die with pods (incident analysis of Airbyte syncs / Argo workflows / dbt runs is painful), there is no fleet-wide view of instance health, and we cannot tell how an update went on a given instance without SSH/kubectl digging. The only existing Grafana lives in the Constructor dev instance, and the way it was installed is questionable — it may need to be reinstalled properly as part of this work.

Goal

A two-tier observability setup:

1. Per-instance stack — each instance runs its own local observability (logs + metrics + alerting), usable by the instance admin on its own.
2. Central monitoring host — a dedicated host/cluster (independent from product instances) that aggregates telemetry from all instances that agree to share it, giving our team a single pane of glass.

Requirements

Scope of instances

• Current: Virtuozzo (insight-v2), dev instances (dev-vhc, insight-dev, …).
• Future client instances must get this out of the box (part of the umbrella chart / standard deployment).
• Sending telemetry to the center is opt-in per instance; expect telemetry categories/levels (e.g., an instance may share only versions + health signals but not full logs). The category model should be designed as part of this task.

What to collect

• Ingestion pipeline logs: Airbyte sync pods, Argo workflows, dbt runs — must survive pod termination.
• Application logs: backend services (identity-resolution, analytics, gateway, connectors, …).
• Infrastructure metrics: CPU/RAM/disk, pod states, ClickHouse/Postgres health, service uptime.
• Deploy/update signals: Helm release status, migration success/failure, deploy annotations on dashboards.

Dashboards (central)

• Fleet version dashboard: which umbrella chart / image versions each instance runs, who is behind.
• Deploy status: how the latest update went on each instance.
• Pipeline health: nightly sync / workflow statuses per instance (what's green, what failed).
• Infra health per instance.
• Reliability KPIs (for Quality: Reliability vector #1786): uptime, share of successful connector syncs, count of failed upgrades/deployments.

Alerting

• Per-instance alerting configured by the instance admin; central alerting configured by the central Grafana admin.
• Supported channels: Zulip, Slack, Email.

Access

• No hard multi-tenancy isolation required in the central stack (single team consumes it); an instance label is enough.

Stack

Loki + Grafana (+ Prometheus-compatible metrics) is the preferred stack, not a hard requirement — DevOps may propose alternatives with justification.

Open questions (to resolve during the task)

• Log retention period and storage backend (local disk vs S3/MinIO).
• Location and sizing of the central monitoring host.
• Log collector choice (Grafana Alloy / promtail / vector / …).
• Transport & auth from instances to the center (egress constraints, TLS, tokens).
• Exact telemetry category definitions.
• What to do with the existing Grafana in the Constructor dev instance (audit → reuse or reinstall).

Deliverables

• Architecture proposal (short design doc) covering the open questions above.
• Central monitoring host up and running (Grafana + Loki + metrics backend).
• Per-instance observability packaged into the standard deployment (umbrella chart component), rolled out to Virtuozzo and dev instances.
• The four central dashboards listed above plus the reliability KPI dashboard.
• Alerting wired to Zulip/Slack/Email.