Skip to content

SSE/live-query yields no events for tables populated by a ClickHouse MATERIALIZED VIEW (only ingest-API writes republish to NATS) #362

Description

@EricAndrechek

Summary

GET /v1/stream?table=T only emits events for rows written via POST /v1/ingest (which publishes to NATS ingest.<table>; the SSE hub subscribes ingest.>). If T is the target table of a ClickHouse MATERIALIZED VIEW — rows land in T via the MV, not the ingest API — no NATS event is published and the stream is silently empty (no error).

Why it surprised us

A natural WaveHouse pattern is: ingest raw rows into a base table, expose a derived/projection table via an MV (e.g. a column-masked positions_public), and stream the projection to public clients. That projection never streams — the base table does, but not the MV target.

(Distinct from the already-tracked row-filter drift in applyStreamPolicy, which is about which columns/rows an event carries. This is about MV-written rows producing no event at all.)

Repro

  1. CREATE TABLE base (...), CREATE TABLE derived (...), CREATE MATERIALIZED VIEW mv TO derived AS SELECT ... FROM base.
  2. GET /v1/stream?table=derived (subscribe).
  3. POST /v1/ingest?table=base a row that the MV projects into derived.
  4. → the row appears in derived via /v1/query, but the derived stream receives nothing.

Impact

Live dashboards built on derived/projection tables silently receive zero updates — easy to misdiagnose as a client/auth problem. We only found it by testing each surface.

Workaround

Stream the base (ingest-API-written) table directly and rely on the JWT column policy to mask columns per role.

Suggested

  • At minimum, document prominently that SSE/live-query only covers ingest-API-written tables (not MV/derived targets).
  • Optionally, a mechanism to surface MV/derived-table changes to the SSE hub (opt-in re-publish of an MV target to NATS, or CDC).

Found building the SCC 2026 tracker pilot against a self-hosted WaveHouse.

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/apiHTTP handlers, routing, middlewarearea/ingestIngest pipeline (Bento, batching, DLQ)enhancementNew feature or request

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    Status
    Backlog

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions