diff --git a/AGENTS.md b/AGENTS.md index 78f20c56..15369cb5 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -59,7 +59,7 @@ The invariant index — what must stay true. Full narrative and rationale live i 10. **Active Sweeper** — purges NATS messages that are both ACKed (written to CH) and older than the gap window; SSE gap-fill uses `DeliverByStartTime`, no in-process ring buffer. 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/admin` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** — a total lockout, so bootstrap from the policy file, never an implicit admin grant. `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`. 12. **Structured queries: column authz fail-closed (security)** — `POST /v1/query?table={table}`: typed AST validated against schema, permission-enforced, timestamp-bucketed for cache, `DefaultMaxRows` (10,000) cap. Every column reference — projection, aggregation args, `filters`, `group_by`, `order_by`, `time_range` — is authorized inside `query.Build` (the single chokepoint that enumerates them all), so no clause can skip the role's `allow_columns`/`deny_columns` check (#223). A `select_all` read by a *column-restricted* role expands to its allowed columns via `policy.AllowedProjection`, never a bare `SELECT *`; *unrestricted*/admin roles keep `SELECT *` (`policy.RestrictsColumns` decides). Omitting `columns` selects nothing (`ErrEmptyProjection` → `200 []`); `["*"]` is the literal column `*` (schema-gated, not a wildcard); a table-granted role with no readable columns fails closed (`ErrNoReadableColumns` → `403`). Structured and live-stream (`filterEventColumns`) reads share the one per-column decision `policy.IsColumnAllowed`, so column visibility can't drift. Preserve when touching `internal/query` or the structured-query handler. Detail: architecture.md § `query/`. -13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). Detail: architecture.md § `pipes/`. +13. **Named query pipes: fail-closed (security)** — pre-defined SQL templates (Tinybird-style) with param binding + caching; `GET/POST /v1/pipes/{name}` sit outside `RequireAdmin`, so per-pipe `allowed_roles` is the *only* execute-path gate, via `policy.RoleAllowed`: exact allowlist membership (no `"*"`), admin always passes, empty/absent role and empty-string entries authorize nobody, and no `allowed_roles` → admin-only. Preserve and exercise via `testutil.RunRoleMatrix` / `StandardRoleMatrix` (see #159). The pipe's SQL is **not** re-checked against the table policy (no row or column filtering) — `allowed_roles` gates execution, but the author is responsible for scoping the data in the pipe SQL and any views it reads. A pipe is **run as-is** — WaveHouse never parses its SQL and never rejects it (a write/DDL or multi-statement pipe runs on behalf of the caller's role, so the author owns scoping it). Cache invalidation derives the pipe's table dependencies from ClickHouse `EXPLAIN QUERY TREE`: a query it can't analyze (a write/DDL pipe, a missing table, an unreachable server) **over-resolves** to every base table so any write evicts it — never an under-resolution — and a resolved dependency whose version can't be reliably maintained (an unfoldable view) is **TTL-floored** (`cache.UnresolvedDepsTTLCap`) rather than trusted to version invalidation. Detail: architecture.md § `pipes/`. 14. **TypeScript SDK** — `@wavehouse/sdk`: zero-dep client, typed query builder, real-time SSE, live queries (incrementable/decomposable/poll aggregation), codegen CLI. The canonical client (see §SDK Sync). 15. **Observability invariants** — stdout always 100% (sampling is OTLP-push-only); WARN+ERROR always export at 100% (a non-configurable floor — don't expose it); gRPC OTel exporters dial lazily so an unreachable collector never blocks startup; the OTel Prometheus exporter uses a **private** `prometheus.Registry`. The OTLP endpoint/TLS/custom-CA/mTLS/headers are delegated to the OpenTelemetry SDK's standard `OTEL_EXPORTER_OTLP_*` env vars — `InitProvider` passes **no** endpoint/header options. Known gap, intentionally not patched in WaveHouse app code: the pinned gRPC logs exporter (`otlploggrpc` v0.19/v0.20) ignores the env TLS-cert vars, so a custom/private CA and mutual TLS apply to traces/metrics but **not** the logs signal (public-CA/system-roots TLS and plaintext still work for logs) — upstream bug open-telemetry/opentelemetry-go#6661. A malformed `OTEL_EXPORTER_OTLP_HEADERS` is logged and skipped by the SDK (fail-soft), not fatal. Preserve when touching the logger/sampler/provider. Detail: architecture.md § `observability/`. 16. **Bearer-token-only CORS posture (security)** — Bearer JWT on every request, no cookies/sessions; `corsMiddleware` deliberately **never** emits `Access-Control-Allow-Credentials` (not needed, and `*` + credentials is a spec violation browsers reject). `cors_allowed_origins` controls who can *read* responses, not cookie scope; CSRF protection is structural. Don't reintroduce cookie auth or `Allow-Credentials` without a design discussion — answers GitHub #29/#30. Code: `internal/api/router.go`. diff --git a/CHANGELOG.md b/CHANGELOG.md index 22af33b7..9efc83c9 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **Named pipes invalidate their cached results on writes to the tables they read, via versioned namespaces and a refresh-time cascade** (`internal/discovery/{discovery,explain}.go`, `internal/api/pipes.go`, `internal/pipes/pipes.go`, `internal/cache/{cache,local,version_manager}.go`, `internal/chsql/nats.go` (moved from `internal/query/ident.go`), `cmd/wavehouse/main.go`, `docs/src/content/docs/{pipes.mdx,architecture.md}`, plus unit + live-ClickHouse tests): closes [#178](https://github.com/Wave-RF/WaveHouse/issues/178). A pipe's table dependencies are resolved by **asking ClickHouse** — `EXPLAIN QUERY TREE` on the bound query the first time it runs (`discovery.ResolveTables`, reading the table list off ClickHouse's own analysis), cached per bound query (parameter combination). WaveHouse never parses pipe SQL itself and never rejects a pipe. `EXPLAIN QUERY TREE` is the pre-optimization tree, so it keeps a table even when the query is metadata-answered (`SELECT count() FROM t` still tracks `t`) or reads a currently-empty table — both of which `system.query_log` and the post-optimization `EXPLAIN PLAN` would drop. It also tracks reads that aren't plain `FROM` clauses: a `joinGet()` of a `Join`-engine table and a `dictGet()` of a dictionary (mapped via `system.dictionaries` to the ClickHouse table backing it). Table-name extraction is robust to `EXPLAIN`-rendering quirks the unit fixtures didn't originally cover: a `FINAL`/`SAMPLE` modifier appends fields after the name (`table_name: db.t, final: 1`) and a string literal can itself contain the text `table_name:` — neither leaks into, nor fabricates, a tracked table — and a `dictGet` against a not-yet-loaded dictionary force-loads it first, since ClickHouse reports an empty `source` for an unloaded dict that would otherwise drop its backing table. A build-tagged differential fuzzer (`internal/discovery/fuzz_deps_test.go`, run with `-tags fuzzdeps` against a local ClickHouse) cross-checks the resolved set for hundreds of nested / `UNION` / parameter-gated queries against a construction-based ground truth. Each resolved table is folded into the cache key as its **own versioned namespace** (a per-name lookup, no per-request graph walk), so a write to it misses the key. Views and materialized views are versioned namespaces too: the schema registry resolves each view's definition (also via EXPLAIN) **once per schema change** into a *cascade* (base table → the views reading it, transitively), pushed into the cache, so a write to a base table also bumps every view over it and evicts a pipe reading the view; a view redefined in place (and the views built on it) is evicted at the next refresh. Resolution is **precise per parameter binding, and otherwise over-resolves — never under**: for a parameter-gated `UNION`, ClickHouse folds the bound predicate so a constant-false arm is pruned (`source=web` depends only on `web_events`, not `mobile_events`; resolved and cached per bound query, and emitting the `wavehouse_pipe_dep_tables_pruned_total` metric), while a query ClickHouse can't analyze (a write/DDL pipe, an unreachable server, a missing table) over-resolves to **all base tables** so any write evicts it. The pruning is conservative — an arm is dropped only when its filter is provably constant-false, so anything unproven stays tracked (over-resolve, never stale). Schema refresh is **atomic** (rebuilt only when content changes) and re-resolves pipes (`PipesHandler.ClearResolvedDeps`). Pipes run the author SQL as-is with no policy row/column filtering (the per-pipe `allowed_roles` gates execution), so every allowed caller shares one cached entry. **Unsupported, and documented as may-go-stale** (run, not blocked): write/ingest pipes, table-function pipes (`s3()`/`numbers()` — their external data isn't a tracked table), and parameterized table names (`FROM {{tbl}}`). Single-database (the configured `clickhouse.database`); a cross-database table is untracked. - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes. - **"Behind a reverse proxy" deployment guide** (`docs/src/content/docs/reverse-proxy.mdx` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `internal/api/stream.go`, `internal/api/stream_test.go`): closes #241. A new Operations page for the common "WaveHouse behind nginx / Caddy / Cloudflare Tunnel" setup, since several behaviors only matter behind a proxy and weren't documented together. Covers: TLS termination (WaveHouse serves plain HTTP and manages no certs); the request-body size limits and the division of responsibility (WaveHouse ships fixed in-code memory-safety backstops — 1 MiB control / 16 MiB ingest — while the proxy is the tunable *outer* limit, so a missing/loose proxy limit can't OOM the server); Server-Sent Events buffering + idle-timeout tuning (WaveHouse sends a `: connected` comment on open plus a periodic `:` keepalive comment so quiet streams survive proxy idle timeouts, [#226](https://github.com/Wave-RF/WaveHouse/issues/226)); the `?token=` / `since` / `Last-Event-ID` forwarding streams need; `X-Forwarded-For` trust (don't expose `:8080` directly — it's honored, so a direct client could spoof it); and which health paths to expose (`/livez`/`/readyz` internal-optional, `/v1/health` must stay public). Ships full example nginx, Caddy, and Cloudflare-Tunnel configs, and is cross-linked from Deployment, Configuration, and the API reference. One small code change lands with it: the SSE endpoint (`GET /v1/stream`) now sets `X-Accel-Buffering: no` so nginx-class proxies stream events without buffering out of the box (nginx strips the header before the client sees it; Caddy/Cloudflare ignore it). The health-probe guidance is also upgraded from "optional" to a recommendation — keep the bare `/livez`/`/readyz`/`/healthz` paths internal (a public `/readyz` turns each hit into a ClickHouse `Ping`) and expose only `/v1/health` publicly. - **Coverage publishing — a self-hosted Go coverage README badge and GitHub Code Quality PR comments** (`.github/workflows/ci.yml`, `.github/actionlint.yaml` (new), `scripts/cov/main.go`, `scripts/ci/publish-badge.sh` (new), `.testcoverage.yml`, `go.mod`/`go.sum`, `README.md`, `AGENTS.md`, `docs/src/content/docs/development.md`): closes #133, now that the repo is public. Two published surfaces, both **non-gating** — `make cov`'s thresholds stay the only merge gate. (1) **README badge**: a new `cov badge` subcommand renders a [shields.io endpoint](https://shields.io/endpoint) JSON for the merged Go total using the *exact* number `threshold.total` gates (same `.testcoverage.yml` excludes), and a new non-gating `badge` job — the sole holder of `contents:write`, running only on trusted main — publishes it to an orphan `badges` branch via `scripts/ci/publish-badge.sh`, which the README reads over `raw.githubusercontent.com` (unrestricted for a public repo). (2) **PR comments**: the `coverage` job converts the merged Go profile to Cobertura (`go tool gocover-cobertura`, a new pinned Go `tool` dependency, with `-ignore-dirs` mirroring the YAML's global excludes) and uploads it to GitHub Code Quality via `actions/upload-code-coverage` (`code-quality: write`); the `github-code-quality[bot]` posts the aggregate + per-file diff-vs-`main` comment. The upload is `continue-on-error` so this public-preview GitHub feature can never red CI, and fork PRs skip it (no `code-quality` token, per GitHub's own guard). `actionlint` doesn't recognize the preview `code-quality` permission scope yet, so a new `.github/actionlint.yaml` suppresses only that one message. Requires the repo's *Settings → Code quality* enablement for the comments to render. Full design in `.github/workflows/README.md` §"Coverage publishing". diff --git a/cmd/wavehouse/main.go b/cmd/wavehouse/main.go index 88406746..24fdafac 100644 --- a/cmd/wavehouse/main.go +++ b/cmd/wavehouse/main.go @@ -188,6 +188,30 @@ func run() int { bootState := api.NewBootState(nil) refreshInterval := time.Duration(cfg.Schema.RefreshInterval) * time.Second registry := discovery.NewSchemaRegistry(chConn, cfg.ClickHouse.Database, refreshInterval, logger) + + // TODO: this is where we can switch between ristretto, redis, tiered (both), etc. + l1, err := cache.NewLocal(cfg.Cache.L1MaxCost) + if err != nil { + logger.Error("cache init", "error", err) + return 1 + } + defer func() { _ = l1.Close() }() + + var pipesHandler *api.PipesHandler + registry.SetOnRefresh(func(snap discovery.DependencySnapshot) { + l1.SetDependents(snap.Cascade) + if len(snap.ChangedViews) > 0 { + nss := make([]cache.Namespace, len(snap.ChangedViews)) + for i, v := range snap.ChangedViews { + nss[i] = cache.Namespace{Table: v} + } + _, _ = l1.Invalidate(ctx, nss) + } + if pipesHandler != nil { + pipesHandler.ClearResolvedDeps() + } + }) + if err := registry.Refresh(ctx); err != nil { logger.Warn("schema discovery failed on boot, retrying in background", "error", err) bootState.Set(fmt.Errorf("schema discovery: %w", err)) @@ -252,16 +276,6 @@ func run() int { } } - // L1 cache only in standalone mode. - l1, err := cache.NewLocal(cfg.Cache.L1MaxCost) - if err != nil { - logger.Error("cache init", "error", err) - return 1 - } - // TODO: eventually this is where we can switch between ristretto, redis, tiered (both), etc - cache := l1 - defer func() { _ = cache.Close() }() - // Policy store (NATS KV + optional file bootstrap). policyStore, err := policy.NewStore(ctx, embeddedMQ.JetStream(), cfg.Policy.FilePath, logger) if err != nil { @@ -291,7 +305,7 @@ func run() int { ingestCleanup, err := ingest.StartIngestWorker( ctx, embeddedMQ.NatsConn(), - cache, + l1, cfg.ClickHouse.Addr, cfg.ClickHouse.HTTPPort, // Uses 8123 by default cfg.ClickHouse.HTTPScheme, @@ -379,6 +393,13 @@ func run() int { return 1 } + // Pipes resolve their cache dependencies through the schema registry's + // dependency tree, so wire the registry beyond NewPipesHandler's core args. + pipesHandler = api.NewPipesHandler(pipesStore, policyStore, chConn, l1, cfg.ClickHouse.QueryTimeout, logger) + pipesHandler.Registry = registry + + structuredQueryHandler := api.NewStructuredQueryHandler(chConn, l1, registry, policyStore, cfg.Cache.TimestampBucketSeconds, cfg.ClickHouse.QueryTimeout, cfg.Query.DefaultMaxRows, logger) + deps := api.Dependencies{ Ingest: ingestHandler, Query: queryHandler, @@ -388,8 +409,8 @@ func run() int { Schema: api.NewSchemaHandler(registry), DLQ: dlqHandler, Policy: api.NewPolicyHandler(policyStore), - Pipes: api.NewPipesHandler(pipesStore, policyStore, chConn, cache, cfg.ClickHouse.QueryTimeout, logger), - StructuredQuery: api.NewStructuredQueryHandler(chConn, cache, registry, policyStore, cfg.Cache.TimestampBucketSeconds, cfg.ClickHouse.QueryTimeout, cfg.Query.DefaultMaxRows, logger), + Pipes: pipesHandler, + StructuredQuery: structuredQueryHandler, AuthMW: authMW, PolicyStore: policyStore, Logger: logger, diff --git a/docs/src/content/docs/architecture.md b/docs/src/content/docs/architecture.md index c7be5082..39719831 100644 --- a/docs/src/content/docs/architecture.md +++ b/docs/src/content/docs/architecture.md @@ -120,7 +120,7 @@ The SSE fan-out primitives shared by the streaming API, factored out of `api/` s ### `ingest/` — Ingest Pipeline, DLQ & Sweeping -- **worker.go** — `StartIngestWorker` launches an ingest pipeline: a JetStream consumer reads from the `WAVEHOUSE` stream via a durable `buffer-consumer` pull subscription, batches events per table, and performs bulk INSERTs to ClickHouse. The pipeline is **insert-only**. The wire format `EventMessage` carries `{table_name, received_timestamp, data}` and nothing else; the worker accepts any table name now (the table name in the NATS subject is `query.SafeEncodeNATS(rawUnsafeTableName)`), then bulk-INSERTs. The embedded NATS server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/admin/query` under the admin role (`policy.admin_role`) — see the Query Path section below; the `/v1/admin/*` `RequireAdmin` middleware enforces the check at the API layer, so a no/invalid-token request (resolved to `default_role`, not admin in a production config) never reaches the proxy. Failed batches are routed to the DLQ (`sendToDLQ`), which republishes the original `EventMessage` envelope to `dlq.{table}` NATS subjects with the failure context in `X-DLQ-*` headers when DLQ is enabled — see [Ingest Pipeline](/ingest-pipeline) for the worker internals. +- **worker.go** — `StartIngestWorker` launches an ingest pipeline: a JetStream consumer reads from the `WAVEHOUSE` stream via a durable `buffer-consumer` pull subscription, batches events per table, and performs bulk INSERTs to ClickHouse. The pipeline is **insert-only**. The wire format `EventMessage` carries `{table_name, received_timestamp, data}` and nothing else; the worker accepts any table name now (the table name in the NATS subject is `chsql.SafeEncodeNATS(rawUnsafeTableName)`), then bulk-INSERTs. The embedded NATS server runs with `DontListen: true` (`internal/mq/embedded.go`), so the only Publishers reachable on the `ingest.>` subjects are in-process Go code — today, only the HTTP `/v1/ingest?table={table}` handler. Non-insert mutations (`DELETE`/`UPDATE`/`TRUNCATE`/…) must go through `POST /v1/admin/query` under the admin role (`policy.admin_role`) — see the Query Path section below; the `/v1/admin/*` `RequireAdmin` middleware enforces the check at the API layer, so a no/invalid-token request (resolved to `default_role`, not admin in a production config) never reaches the proxy. Failed batches are routed to the DLQ (`sendToDLQ`), which republishes the original `EventMessage` envelope to `dlq.{table}` NATS subjects with the failure context in `X-DLQ-*` headers when DLQ is enabled — see [Ingest Pipeline](/ingest-pipeline) for the worker internals. - **types.go** — `EventMessage` struct (TableName, ReceivedTimestamp, Data) and `BufferConsumerName` constant, shared across API handlers and the ingest pipeline. - **sweeper.go** — `Sweeper` implements the Active Sweeper pattern. It runs every minute and purges NATS JetStream messages that are **both** ACKed by the buffer consumer (written to ClickHouse) **and** older than the configurable gap window. @@ -145,7 +145,7 @@ The package's design invariants — stdout always 100%, WARN+ERROR always export ### `pipes/` — Named Query Pipes -- **pipes.go** — `NamedQuery` type with SQL template and parameter definitions, `Store` backed by NATS KV bucket `WAVEHOUSE_PIPES`. Supports `.sql` file directory bootstrap. `BindParams()` resolves `{{param}}` / `{{param:default}}` placeholders by inlining escaped literal values into the SQL (strings single-quote-escaped; arrays rendered as escaped `(…)` `IN`-lists). A non-scalar value with no SQL form (a JSON object, or an empty array) is rejected rather than emitted raw. +- **pipes.go** — `NamedQuery` type with SQL template and parameter definitions, `Store` backed by NATS KV bucket `WAVEHOUSE_PIPES`. Supports `.sql` file directory bootstrap. `BindParams()` resolves `{{param}}` / `{{param:default}}` placeholders by inlining escaped literal values into the SQL (strings single-quote-escaped; arrays rendered as escaped `(…)` `IN`-lists). A non-scalar value with no SQL form (a JSON object, or an empty array) is rejected rather than emitted raw. The Store does **no** SQL analysis and rejects nothing — a pipe's table dependencies are resolved by asking ClickHouse: `api.PipesHandler.resolveDeps` runs `EXPLAIN QUERY TREE` on the bound query (the first time the pipe runs) and reads the table list off ClickHouse's own analysis, cached per bound query (parameter combination). Each resolved table is folded into the cache key as its own versioned namespace; a query ClickHouse can't analyze (a write/DDL pipe, an unreachable server) over-resolves to all base tables (`SchemaRegistry.AllBaseTables`) — never under-resolves. A resolved dependency whose version the cascade can't reliably maintain — an unfoldable view (unparsed definition, or a cross-database/unknown source), per `SchemaRegistry.IsKnown` — is instead TTL-floored (`cache.UnresolvedDepsTTLCap`, applied at the pipe `Set`) rather than trusted to version invalidation. `EXPLAIN QUERY TREE` is the pre-optimization tree, so it keeps a table even when the query is metadata-answered (`count()`) and keeps every parameter-gated `UNION` branch, both of which `system.query_log` would drop. Views and materialized views are versioned namespaces too: the schema registry resolves each view's definition (also via `discovery.ResolveTables`/EXPLAIN) once per refresh into a base-table→view *cascade* (`SchemaRegistry.Dependents`, pushed into the cache via `SetOnRefresh`), so a write to a base table also bumps every view over it and evicts a pipe reading that view; `PipesHandler.ClearResolvedDeps` re-resolves pipes on each refresh. Table functions (`s3()`) and parameterized table names are unsupported — those pipes run but may go stale (documented). The pipe handler applies no policy row/column filtering to the author SQL (see pipes docs). ### `query/` — Structured Query Engine diff --git a/docs/src/content/docs/pipes.mdx b/docs/src/content/docs/pipes.mdx index dc9f262c..84c654c1 100644 --- a/docs/src/content/docs/pipes.mdx +++ b/docs/src/content/docs/pipes.mdx @@ -119,7 +119,7 @@ Pipe execution is gated by `allowed_roles` — a plain allowlist, evaluated inde To make a pipe public, list the role that `default_role` resolves to. For example, with `default_role: viewer`, a pipe with `"allowed_roles": ["viewer"]` is reachable with no token at all. -This is the *only* authorization check on the execute path — pipes deliberately sit outside the `/v1/admin/*` gate so non-admin callers can run them. The pipe's SQL is not re-checked against the table policy's column or row rules, so **scope the data in the pipe's SQL itself** (or via a claim-templated predicate) rather than assuming policy column masking applies. +This is the *only* authorization check on the execute path — pipes deliberately sit outside the `/v1/admin/*` gate so non-admin callers can run them. The pipe's SQL is **not** re-checked against the table policy's column or row rules — neither the projection nor the rows are filtered by policy — so **scope the data in the pipe's SQL itself** (and in any views it reads) rather than assuming policy column masking or row security applies. Row security can't be applied reliably to opaque SQL that may read arbitrary views (a view over a table without the filter column can't be filtered, and would fail silently), so it is deliberately left to the pipe author. ## Creating and managing pipes @@ -171,6 +171,22 @@ curl -X POST http://localhost:8080/v1/pipes/top_pages \ The response is a JSON array of rows. Results flow through the shared in-process L1 cache (Ristretto) with singleflight coalescing, so concurrent identical calls hit ClickHouse once; an `X-Cache: HIT` or `X-Cache: MISS` header tells you which path served the response. +A cached pipe result is keyed by its SQL and parameter values (every allowed caller shares one entry — pipes apply no per-role filtering), and is **invalidated by writes to the tables it reads**. WaveHouse never parses your SQL to find those tables — it **asks ClickHouse**, running `EXPLAIN QUERY TREE` the first time a pipe runs with a given set of parameter values and reading the table list off ClickHouse's own analysis. That result is cached and reused for the same parameters, so the extra round-trip happens once per parameter combination. Working off the pre-optimization query tree, it captures a table even when the query is answered from metadata (`SELECT count() FROM t` still tracks `t`) or reads a currently-empty table — neither of which a runtime signal like `system.query_log` or the post-optimization `EXPLAIN PLAN` would report. It also tracks reads that aren't plain `FROM` clauses: a `joinGet()` of a `Join`-engine table, and a `dictGet()` of a dictionary (mapped to the ClickHouse table backing it). + +Each table EXPLAIN reports carries its **own cache version**, folded into the key — a write to it evicts the result. Views and materialized views are versioned just like base tables. The link between a view and its base is precomputed: the schema registry resolves each view's definition (again via EXPLAIN) **once when the schema is discovered** into a *cascade* — each base table mapped to the views that read it, recursively through chained views — so a write to a base table also bumps every dependent view's version, evicting a pipe reading the view. (For a materialized view, the source table is what ingest bumps and from which ClickHouse maintains the view.) A view redefined in place (same sources, new body) is evicted on the next schema refresh, along with the views built on top of it. + +Resolution is **precise per parameter binding, and where it can't be precise it over-resolves — never under**. For a parameter-gated `UNION`, ClickHouse folds the bound predicate: with `?source=web`, the `… UNION ALL … WHERE {{source}} = 'mobile'` arm becomes constant-false and reads nothing, so `mobile_events` is dropped from the dependency set — a write to it does **not** evict the `source=web` result (that arm can't read `mobile_events` regardless of its data, so this is precise, not stale). The `source=mobile` result is resolved and cached separately, depending only on `mobile_events`. The pruning is deliberately conservative: it drops an arm only when ClickHouse proves its filter constant-false, so anything it can't prove dead stays tracked. And if WaveHouse can't get an answer from ClickHouse at all — the query references a table that doesn't exist yet, or ClickHouse is briefly unreachable — the pipe **over-resolves to every table** so any write evicts it: again never stale, just coarse until it re-resolves. + +:::caution[Unsupported pipes — run, but may go stale] +Pipes are stored and run **as-is** — WaveHouse never rejects one. But some pipes can't be cache-tracked, and **their cached results may go stale**. WaveHouse doesn't block them; it's the developer's call: + +- **Writes / DDL** (`INSERT`, `CREATE`, `DROP`, …). A pipe is meant to be a read. A write pipe runs but doesn't go through the ingest path, and its result over-resolves (any write evicts it) — not a supported pattern. +- **Table functions** (`s3()`, `numbers()`, `url()`, …). Their data lives outside ClickHouse's tables, so there's no table to version. A pipe reading `s3()` is cached and only its non-function tables are tracked; a change to the external source won't evict it until the TTL expires. +- **Parameterized table names** (`FROM {{tbl}}`, `FROM events_{{year}}`). The table varies per parameter value, which WaveHouse can't track without parsing the SQL — unsupported. + +Pipe dependencies re-resolve on a **schema refresh** (`POST /v1/schema/refresh`, plus the periodic tick), so after you create/edit a pipe or change the schema, a refresh re-pins everything against the current ClickHouse state. +::: + | Status | Body | Cause | | ------ | ---- | ----- | | 404 | `{"error":"pipe not found"}` | No pipe registered under that name | diff --git a/internal/api/boot_chain_test.go b/internal/api/boot_chain_test.go index 4f5b22ad..2dec2db7 100644 --- a/internal/api/boot_chain_test.go +++ b/internal/api/boot_chain_test.go @@ -112,9 +112,5 @@ func TestBoot_Chain_DegradedThenRecovers(t *testing.T) { handler.Liveness(rec, httptest.NewRequestWithContext(context.Background(), http.MethodGet, "/livez", nil)) assert.Equal(t, http.StatusOK, rec.Code) assert.Contains(t, rec.Body.String(), `"status":"ok"`) - - // Sanity: exactly four Query calls — the initial sync Refresh (1) plus - // the retry loop's three attempts (2,3 fail; 4 succeeds). An off-by-one - // in the loop's success short-circuit would show up here. - assert.Equal(t, int32(4), conn.calls.Load(), "expected 4 Query calls total") + assert.Equal(t, int32(5), conn.calls.Load(), "expected 5 Query calls total") } diff --git a/internal/api/dlq.go b/internal/api/dlq.go index 5d57c91b..323e7e37 100644 --- a/internal/api/dlq.go +++ b/internal/api/dlq.go @@ -7,8 +7,8 @@ import ( "net/http" "strings" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/mq" - "github.com/Wave-RF/WaveHouse/internal/query" "github.com/nats-io/nats.go/jetstream" ) @@ -37,7 +37,7 @@ func (h *DLQHandler) Stats(w http.ResponseWriter, r *http.Request) { subjectFilter := ">" tableFilter := r.URL.Query().Get("table") if tableFilter != "" { - subjectFilter = "dlq." + query.SafeEncodeNATS(tableFilter) + subjectFilter = "dlq." + chsql.SafeEncodeNATS(tableFilter) } info, err := stream.Info(r.Context(), jetstream.WithSubjectFilter(subjectFilter)) @@ -49,7 +49,7 @@ func (h *DLQHandler) Stats(w http.ResponseWriter, r *http.Request) { tables := make(map[string]uint64) for subject, count := range info.State.Subjects { // TODO: do we need to break out scopes here? - decodedSubject, err := query.SafeDecodeNATS(strings.TrimPrefix(subject, "dlq.")) + decodedSubject, err := chsql.SafeDecodeNATS(strings.TrimPrefix(subject, "dlq.")) if err != nil { continue } diff --git a/internal/api/ingest.go b/internal/api/ingest.go index 4d361343..080657fd 100644 --- a/internal/api/ingest.go +++ b/internal/api/ingest.go @@ -12,12 +12,12 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/auth" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/dedupe" "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/ingest" "github.com/Wave-RF/WaveHouse/internal/mq" "github.com/Wave-RF/WaveHouse/internal/policy" - "github.com/Wave-RF/WaveHouse/internal/query" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" @@ -409,9 +409,9 @@ func (h *IngestHandler) processRecord( return false, nil, &requestAbort{Status: http.StatusInternalServerError, Message: "marshal failed"} } - subject := "ingest." + query.SafeEncodeNATS(table) + subject := "ingest." + chsql.SafeEncodeNATS(table) if scope != "" { - subject += "." + query.SafeEncodeNATS(scope) + subject += "." + chsql.SafeEncodeNATS(scope) } h.logger.DebugContext(ctx, "publishing event to NATS", "subject", subject, "table", table, "scope", scope) diff --git a/internal/api/pipe_deps_test.go b/internal/api/pipe_deps_test.go new file mode 100644 index 00000000..3f825689 --- /dev/null +++ b/internal/api/pipe_deps_test.go @@ -0,0 +1,117 @@ +package api + +import ( + "context" + "errors" + "sort" + "testing" + "time" + + "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "github.com/Wave-RF/WaveHouse/internal/discovery" + "github.com/Wave-RF/WaveHouse/internal/testutil" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func pipeDepNames(deps []cache.Namespace) []string { + out := make([]string, len(deps)) + for i, d := range deps { + out[i], _ = chsql.SafeDecodeNATS(d.Table) + } + sort.Strings(out) + return out +} + +// resolveDeps asks ClickHouse (EXPLAIN QUERY TREE) which tables a pipe reads — +// mocked here via resolveTablesFn so the unit test needs no server. It covers the +// fold, the per-pipe cache (EXPLAIN runs once), the all-tables over-resolve fallback +// when ClickHouse can't analyze the query, and ClearResolvedDeps re-resolution. +func TestResolveDeps_Explain(t *testing.T) { + ctx := context.Background() + reg := discovery.NewSchemaRegistryForTest( + []*discovery.TableSchema{{Name: "events"}, {Name: "users"}, {Name: "orders"}}, nil) + lc, _ := cache.NewLocal(1 << 20) + defer func() { _ = lc.Close() }() + + calls := 0 + h := &PipesHandler{Registry: reg, Cache: lc, logger: testutil.NopLogger(), pipeDeps: map[string]*resolvedDeps{}} + h.resolveTablesFn = func(_ context.Context, sql string) ([]string, error) { + calls++ + switch sql { + case "SELECT * FROM events": + return []string{"events"}, nil + case "SELECT * FROM events JOIN users ON 1": + return []string{"events", "users"}, nil + case "INSERT INTO events VALUES (1)": // a write → EXPLAIN errors + return nil, errors.New("syntax error") + case "SELECT * FROM mystery": // resolves to a name the registry doesn't know + return []string{"mystery_view"}, nil + default: + return []string{}, nil + } + } + + // resolves to the reported tables; all known → not TTL-floored + deps, unresolved := h.resolveDeps(ctx, "SELECT * FROM events") + assert.Equal(t, []string{"events"}, pipeDepNames(deps)) + assert.False(t, unresolved, "a known base table must not floor the TTL") + + // cached: a second call with the same bound query doesn't re-run EXPLAIN + before := calls + _, _ = h.resolveDeps(ctx, "SELECT * FROM events") + assert.Equal(t, before, calls, "resolveTablesFn must be cached per bound query") + + // multi-table + deps, _ = h.resolveDeps(ctx, "SELECT * FROM events JOIN users ON 1") + assert.Equal(t, []string{"events", "users"}, pipeDepNames(deps)) + + // fallback: an un-analyzable query over-resolves to ALL base tables (never stale, + // and every base table is version-maintained, so it is NOT TTL-floored) + deps, unresolved = h.resolveDeps(ctx, "INSERT INTO events VALUES (1)") + assert.Equal(t, []string{"events", "orders", "users"}, pipeDepNames(deps)) + assert.False(t, unresolved, "the all-base-tables over-resolve must not floor the TTL") + + // a resolved dep the registry doesn't know (an unfoldable view) → TTL-floored + deps, unresolved = h.resolveDeps(ctx, "SELECT * FROM mystery") + assert.Equal(t, []string{"mystery_view"}, pipeDepNames(deps)) + assert.True(t, unresolved, "an unknown/unfoldable dep must floor the TTL") + + // ClearResolvedDeps forces re-resolution + callsBefore := calls + h.ClearResolvedDeps() + _, _ = h.resolveDeps(ctx, "SELECT * FROM events") + assert.Greater(t, calls, callsBefore, "ClearResolvedDeps must force re-resolution") + + // no registry → tracking disabled (TTL-only), no deps + hNil := &PipesHandler{Cache: lc, logger: testutil.NopLogger(), pipeDeps: map[string]*resolvedDeps{}} + nilDeps, nilUnresolved := hNil.resolveDeps(ctx, "SELECT 1") + assert.Empty(t, nilDeps) + assert.False(t, nilUnresolved) +} + +// End-to-end through the real LocalCache: a write to a resolved table evicts the +// pipe; an unrelated write does not. +func TestResolveDeps_Explain_Invalidation(t *testing.T) { + ctx := context.Background() + reg := discovery.NewSchemaRegistryForTest( + []*discovery.TableSchema{{Name: "events"}, {Name: "users"}}, nil) + lc, _ := cache.NewLocal(1 << 20) + defer func() { _ = lc.Close() }() + lc.SetDependents(reg.Dependents()) + h := &PipesHandler{Registry: reg, Cache: lc, logger: testutil.NopLogger(), pipeDeps: map[string]*resolvedDeps{}} + h.resolveTablesFn = func(_ context.Context, _ string) ([]string, error) { return []string{"events"}, nil } + + ns := func(n string) cache.Namespace { return cache.Namespace{Table: chsql.SafeEncodeNATS(n)} } + evicts := func(write string) bool { + deps, _ := h.resolveDeps(ctx, "SELECT * FROM events") + require.NoError(t, lc.Set(ctx, "sha", deps, []byte("v"), time.Hour)) + lc.Wait() + _, _ = lc.Invalidate(ctx, []cache.Namespace{ns(write)}) + d, _, _ := lc.Get(ctx, "sha", deps) + return d == nil + } + assert.True(t, evicts("events"), "write to the resolved table evicts") + assert.False(t, evicts("users"), "unrelated write does not evict") +} diff --git a/internal/api/pipes.go b/internal/api/pipes.go index eac7b853..901a4168 100644 --- a/internal/api/pipes.go +++ b/internal/api/pipes.go @@ -5,17 +5,54 @@ import ( "encoding/json" "log/slog" "net/http" + "sync" "time" "github.com/ClickHouse/clickhouse-go/v2/lib/driver" "github.com/Wave-RF/WaveHouse/internal/auth" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/pipes" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/go-chi/chi/v5" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/metric" "golang.org/x/sync/singleflight" ) +const maxPipeDeps = 50000 + +var ( + pipeFallbackCounter, _ = otel.Meter("wavehouse-pipes").Int64Counter( + "wavehouse_pipe_dep_resolution_fallback_total", + metric.WithDescription("Pipe executions served while over-resolving to all base tables because ClickHouse couldn't analyze the query (a write/DDL pipe, a missing table, an unreachable server)"), + ) + pipeUnresolvedCounter, _ = otel.Meter("wavehouse-pipes").Int64Counter( + "wavehouse_pipe_dep_unresolved_total", + metric.WithDescription("Pipe executions TTL-floored because a resolved dependency can't be reliably version-invalidated (an unfoldable view, or a cross-database/unknown name)"), + ) +) + +type resolvedDeps struct { + tables []string + // fallback is set when EXPLAIN couldn't be used (a write/DDL pipe, a + // missing table, an unreachable ClickHouse), in which case the pipe OVER-resolves + // to every base table so any write evicts it — never an under-resolution. Cached + // per pipe SQL template and cleared on schema refresh. + fallback bool + // unresolved is set when EXPLAIN succeeded but at least one resolved dependency + // can't be reliably version-invalidated — an unfoldable view (unparsed + // definition, or a cross-database/unknown source) or an unknown name, per + // SchemaRegistry.IsKnown. The refresh-time cascade never bumps such a name's + // version, so trusting it would serve stale; the Execute path TTL-floors the + // result instead (cache.UnresolvedDepsTTLCap). A table function or cross-database + // table referenced directly is absent from tables entirely (not flagged here) and + // stays documented-stale. Distinct from fallback, whose all-base-tables set is + // fully known and version-maintained. + unresolved bool +} + // PipesHandler handles named query pipe endpoints. type PipesHandler struct { Store *pipes.Store @@ -25,6 +62,10 @@ type PipesHandler struct { sf singleflight.Group maxQueryTimeout time.Duration logger *slog.Logger + Registry *discovery.SchemaRegistry + resolveTablesFn func(ctx context.Context, sql string) ([]string, error) + pipeDeps map[string]*resolvedDeps + pipeDepsMu sync.Mutex // maxRequestBytes optionally overrides the default inbound request body // cap (maxControlBodyBytes) for the body-decoding paths (Put, Execute). @@ -35,7 +76,21 @@ type PipesHandler struct { } func NewPipesHandler(store *pipes.Store, policyStore *policy.Store, conn driver.Conn, c cache.Cache, queryTimeout time.Duration, logger *slog.Logger) *PipesHandler { - return &PipesHandler{Store: store, PolicyStore: policyStore, CHConn: conn, Cache: c, maxQueryTimeout: queryTimeout, logger: logger} + h := &PipesHandler{ + Store: store, + PolicyStore: policyStore, + CHConn: conn, + Cache: c, + maxQueryTimeout: queryTimeout, + logger: logger, + pipeDeps: make(map[string]*resolvedDeps), + } + // Resolve a pipe's tables by asking ClickHouse (EXPLAIN QUERY TREE). Overridable + // in tests; the database comes from the Registry (set by main post-construction). + h.resolveTablesFn = func(ctx context.Context, sql string) ([]string, error) { + return discovery.ResolveTables(ctx, h.CHConn, h.Registry.Database(), sql) + } + return h } // List returns all named queries (admin endpoint). @@ -168,14 +223,12 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { return } - // Cache. A pipe can read several tables, but the current pipe impl doesn't - // expose its table/scope dependencies, so we pass no deps: the result is keyed - // by sha alone (TTL-only) and the ingest worker cannot version-invalidate it. - // TODO: once pipes expose their tables/scopes, pass them as deps here so writes - // invalidate cached pipe results. cacheKey := queryCacheKey(sql, params) + + deps, depsUnresolved := h.resolveDeps(r.Context(), sql) + if h.Cache != nil { - if data, _, err := h.Cache.Get(r.Context(), cacheKey, nil); err == nil && data != nil { + if data, _, err := h.Cache.Get(r.Context(), cacheKey, deps); err == nil && data != nil { w.Header().Set("Content-Type", "application/json") w.Header().Set("X-Cache", "HIT") _, _ = w.Write(data) @@ -183,13 +236,11 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { } } - // Execute with singleflight. v, err, _ := h.sf.Do(cacheKey, func() (interface{}, error) { queryCtx, cancel := context.WithTimeout(r.Context(), h.maxQueryTimeout) defer cancel() start := time.Now() - rows, err := executeCHQuery(queryCtx, h.CHConn, sql, params) queryDuration := time.Since(start) if err != nil { @@ -203,10 +254,15 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { return nil, err } - ttl := cache.QueryTimeToTTL(queryDuration) - if h.Cache != nil { - _ = h.Cache.Set(r.Context(), cacheKey, nil, data, ttl) + ttl := cache.QueryTimeToTTL(queryDuration) + if depsUnresolved { + // A resolved dependency can't be reliably version-invalidated (an + // unfoldable view): floor the TTL so the result self-expires rather + // than serving stale on a write the cascade never bumps. + ttl = min(ttl, cache.UnresolvedDepsTTLCap) + } + _ = h.Cache.Set(r.Context(), cacheKey, deps, data, ttl) } return data, nil }) @@ -219,3 +275,96 @@ func (h *PipesHandler) Execute(w http.ResponseWriter, r *http.Request) { w.Header().Set("X-Cache", "MISS") _, _ = w.Write(v.([]byte)) } + +// resolveDeps returns the cache namespaces a pipe's result depends on, by asking +// ClickHouse which tables the bound query reads (EXPLAIN QUERY TREE, via +// resolveTablesFn), cached per pipe SQL template. A table EXPLAIN reports is folded +// as its own versioned namespace; a write to it — or, for a view, a write to the +// view's base via the cascade — evicts the result. When ClickHouse can't analyze the +// query (a write/DDL pipe, a missing table, an unreachable server) the pipe +// OVER-resolves to every base table, so any write evicts it: coarse, but never a +// stale read. A table function (s3()/numbers()) or a cross-database table is simply +// absent from the list — those pipes are unsupported and may go stale (documented). +// +// The second return reports whether the result must be TTL-floored: true when a +// resolved dependency can't be reliably version-invalidated (an unfoldable view or +// unknown name — see resolvedDeps.unresolved), so the caller caps the cache TTL +// rather than trust an unmaintained version. +// +// TODO: impl scope (per-tenant cache invalidation) — each dep is whole-table for now. +func (h *PipesHandler) resolveDeps(ctx context.Context, boundSQL string) ([]cache.Namespace, bool) { + if h.Registry == nil { + return nil, false // dependency tracking disabled (no schema registry): TTL-only + } + + h.pipeDepsMu.Lock() + rd, ok := h.pipeDeps[boundSQL] + h.pipeDepsMu.Unlock() + if !ok { + rd = h.resolvePipe(ctx, boundSQL) + h.pipeDepsMu.Lock() + // Bound the per-bound-query cache: a high-cardinality parameter could + // otherwise grow it without limit between schema refreshes. On overflow drop + // the whole map; entries simply re-resolve (one EXPLAIN) on next execution. + if len(h.pipeDeps) >= maxPipeDeps { + h.pipeDeps = make(map[string]*resolvedDeps) + } + h.pipeDeps[boundSQL] = rd + h.pipeDepsMu.Unlock() + } + + names := rd.tables + // fallback and unresolved are mutually exclusive: fallback is set when EXPLAIN + // can't run (resolvePipe returns early), unresolved only when it succeeds. + switch { + case rd.fallback: + names = h.Registry.AllBaseTables() // over-resolve: any write evicts + pipeFallbackCounter.Add(ctx, 1) + case rd.unresolved: + pipeUnresolvedCounter.Add(ctx, 1) + } + deps := make([]cache.Namespace, 0, len(names)) + for _, name := range names { + deps = append(deps, cache.Namespace{Table: chsql.SafeEncodeNATS(name)}) + } + return deps, rd.unresolved +} + +// resolvePipe asks ClickHouse for the bound query's tables. An error (a write/DDL +// pipe, a missing table, an unreachable ClickHouse) yields a fallback marker so the +// caller over-resolves rather than risk an under-resolution. +func (h *PipesHandler) resolvePipe(ctx context.Context, boundSQL string) *resolvedDeps { + if h.resolveTablesFn == nil { + return &resolvedDeps{fallback: true} + } + rctx, cancel := context.WithTimeout(ctx, 5*time.Second) + defer cancel() + tables, err := h.resolveTablesFn(rctx, boundSQL) + if err != nil { + h.logger.DebugContext(ctx, "pipe dependency resolution failed; over-resolving to all tables", "error", err) + return &resolvedDeps{fallback: true} + } + rd := &resolvedDeps{tables: tables} + // A resolved dep whose version isn't reliably maintained (an unfoldable view, an + // unknown name) makes the whole result untrustworthy to version invalidation, so + // mark it for a TTL floor. Registry is non-nil here (resolveDeps guards it). + if h.Registry != nil { + for _, t := range tables { + if !h.Registry.IsKnown(t) { + rd.unresolved = true + h.logger.DebugContext(ctx, "pipe dependency can't be reliably version-invalidated; TTL-flooring result", "table", t) + break + } + } + } + return rd +} + +// ClearResolvedDeps drops every pipe's cached table resolution so each re-resolves +// against the current schema on its next execution. main wires this to a schema +// refresh, so a developer's "edit pipe/schema -> refresh" picks up the change. +func (h *PipesHandler) ClearResolvedDeps() { + h.pipeDepsMu.Lock() + h.pipeDeps = make(map[string]*resolvedDeps) + h.pipeDepsMu.Unlock() +} diff --git a/internal/api/stream.go b/internal/api/stream.go index 37b4caa0..c85a3da1 100644 --- a/internal/api/stream.go +++ b/internal/api/stream.go @@ -8,10 +8,10 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/auth" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/ingest" "github.com/Wave-RF/WaveHouse/internal/mq" "github.com/Wave-RF/WaveHouse/internal/policy" - "github.com/Wave-RF/WaveHouse/internal/query" "github.com/Wave-RF/WaveHouse/internal/stream" "github.com/nats-io/nats.go/jetstream" ) @@ -53,9 +53,9 @@ func (h *StreamHandler) Handle(w http.ResponseWriter, r *http.Request) { // TODO: impl scope scope := "" - topic := "ingest." + query.SafeEncodeNATS(table) + topic := "ingest." + chsql.SafeEncodeNATS(table) if scope != "" { - topic += "." + query.SafeEncodeNATS(scope) + topic += "." + chsql.SafeEncodeNATS(scope) } w.Header().Set("Content-Type", "text/event-stream") diff --git a/internal/api/structured_query.go b/internal/api/structured_query.go index 3564cf35..f4d364bd 100644 --- a/internal/api/structured_query.go +++ b/internal/api/structured_query.go @@ -12,6 +12,7 @@ import ( "github.com/ClickHouse/clickhouse-go/v2/lib/driver" "github.com/Wave-RF/WaveHouse/internal/auth" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/discovery" "github.com/Wave-RF/WaveHouse/internal/policy" "github.com/Wave-RF/WaveHouse/internal/query" @@ -150,12 +151,12 @@ func (h *StructuredQueryHandler) Handle(w http.ResponseWriter, r *http.Request) // TODO: impl scope scope := "" - safeTableName := query.SafeEncodeNATS(table) + safeTableName := chsql.SafeEncodeNATS(table) // A structured query reads one table, so it depends on a single namespace. // Encode the scope the way the ingest worker does (worker.go handleSuccess) so // the read and invalidation sides build identical namespace keys once scope is // implemented; SafeEncodeNATS("") is "", so this is a no-op while scope is empty. - deps := []cache.Namespace{{Table: safeTableName, Scope: query.SafeEncodeNATS(scope)}} + deps := []cache.Namespace{{Table: safeTableName, Scope: chsql.SafeEncodeNATS(scope)}} // Try cache. if h.Cache != nil { diff --git a/internal/cache/cache.go b/internal/cache/cache.go index 90401d62..df2eeff1 100644 --- a/internal/cache/cache.go +++ b/internal/cache/cache.go @@ -12,8 +12,9 @@ type Cache interface { // structured query, several for a pipe). Returns nil, 0, nil on miss. Get(ctx context.Context, sha string, deps []Namespace) ([]byte, time.Duration, error) - // TODO: TTL should be set based on query execution time - // Set stores a query result keyed by sha + its dependency namespaces. + // Set stores a query result keyed by sha + its dependency namespaces. Callers + // derive ttl from query execution time via QueryTimeToTTL; tuning that curve + // with real-world metrics is the remaining work (see QueryTimeToTTL). Set(ctx context.Context, sha string, deps []Namespace, value []byte, ttl time.Duration) error // TODO: option to prefetch pipes when invalidated? @@ -22,15 +23,33 @@ type Cache interface { // Invalidate bumps the version for each namespace, orphaning every cached query // that depends on it. A namespace with an empty Scope bumps the whole table // (every scope); a non-empty Scope bumps just that scope plus the whole-table - // view. Returns the number of namespaces processed. + // view. A write also fans out to the namespace's dependent views (SetDependents). + // Returns the number of namespaces processed. Invalidate(ctx context.Context, namespaces []Namespace) (uint64, error) + // SetDependents installs the base-table -> dependent-view cascade that Invalidate + // fans out through, so a write to a base table also evicts pipes reading a view + // over it. Both sides are NATS-encoded. The schema registry pushes a fresh map on + // every content-changed refresh; replaces any prior one. + SetDependents(dependents map[string][]string) + // TODO: for local cache, we can just store the versions in memory, but for distributed/L2 cache, we will need to be able to either have stored procedures/pipelines etc to query them and attach them to a query, or sync them to each edge api server. // Close releases resources. Close() error } +// UnresolvedDepsTTLCap bounds the lifetime of a pipe result whose dependencies +// could not be fully resolved/proven at definition time (an unknown or +// not-yet-discovered table, an un-parseable view definition, a cross-database +// source, or a legacy pipe whose SQL no longer parses). Such an entry is NOT +// reliably version-invalidatable, so it must self-expire quickly instead. It is +// applied as min(QueryTimeToTTL, cap) at the pipe Set call site — distinct from +// QueryTimeToTTL (the cost-derived happy-path TTL), so the structured-query path, +// whose single dependency always resolves, is unaffected. A fixed in-code +// backstop, not a tuning knob; promote to config only if real data warrants. +const UnresolvedDepsTTLCap = 10 * time.Second + // our tunable function to determine a cache entry's TTL based on how long its queryTime took func QueryTimeToTTL(queryTime time.Duration) time.Duration { // need more real-world data/metrics in order to better refine this, and likely an override option for easy local testing per-deployment diff --git a/internal/cache/local.go b/internal/cache/local.go index 8d6053d9..c9ed3cc2 100644 --- a/internal/cache/local.go +++ b/internal/cache/local.go @@ -58,22 +58,43 @@ func (l *LocalCache) Set(_ context.Context, sha string, deps []Namespace, value // scope at once); a non-empty Scope bumps just that scope plus the whole-table // view. Returns the number of namespaces processed. // -// This bumps exactly what it's given. A whole-table bump already subsumes every -// per-scope bump for the same table (the table version is embedded in every -// namespace key), so a caller that knows a whole-table bump is coming should drop -// the now-redundant scope entries itself — the ingest worker does this as it -// builds the batch, where it already loops once and knows it's a single table. +// Each written table also fans out to its dependent views (SetDependents): a write +// to a base table whole-table-bumps every view reading it, so a pipe that folds a +// VIEW's namespace directly is evicted without the read path ever walking the view +// graph. The fan-out is whole-table (the view's result depends on the table, not a +// scope) and deduped, so a view fed by several written tables bumps once. +// +// This bumps exactly what it's given (plus the fan-out). A whole-table bump already +// subsumes every per-scope bump for the same table (the table version is embedded +// in every namespace key), so a caller that knows a whole-table bump is coming +// should drop the now-redundant scope entries itself — the ingest worker does this +// as it builds the batch, where it already loops once and knows it's a single table. func (l *LocalCache) Invalidate(_ context.Context, namespaces []Namespace) (uint64, error) { + bumpedViews := make(map[string]struct{}) for _, ns := range namespaces { if ns.Scope == "" { l.versionManager.BumpTable(ns.Table) } else { l.versionManager.BumpNamespace(ns.Table, ns.Scope) } + for _, view := range l.versionManager.DependentsOf(ns.Table) { + if _, done := bumpedViews[view]; done { + continue + } + bumpedViews[view] = struct{}{} + l.versionManager.BumpTable(view) + } } return uint64(len(namespaces)), nil } +// SetDependents installs the base-table -> dependent-view cascade used by +// Invalidate to fan a base-table write out to the views reading it. The schema +// registry pushes a fresh map on every content-changed refresh. +func (l *LocalCache) SetDependents(dependents map[string][]string) { + l.versionManager.SetDependents(dependents) +} + // Wait blocks until all buffered writes have been applied. // Exposed for testing; production callers rarely need this. func (l *LocalCache) Wait() { diff --git a/internal/cache/local_test.go b/internal/cache/local_test.go index 47c01fcb..c878ccab 100644 --- a/internal/cache/local_test.go +++ b/internal/cache/local_test.go @@ -134,6 +134,42 @@ func TestLocalCache_Invalidate(t *testing.T) { assert.Zero(t, ttlAfter) } +// SetDependents installs the base-table -> dependent-view cascade: a write to a +// base table must orphan an entry keyed by a VIEW namespace (what a pipe reading +// the view folds), even though the view itself is never written. A write to an +// unrelated table must not. +func TestLocalCache_Invalidate_CascadesToDependentViews(t *testing.T) { + t.Parallel() + c, err := NewLocal(1 << 20) + require.NoError(t, err) + defer func() { _ = c.Close() }() + + ctx := context.Background() + // A pipe reads a view, so its result is keyed by the VIEW's namespace. + viewDeps := []Namespace{{Table: "page_views_mv"}} + c.SetDependents(map[string][]string{"page_views": {"page_views_mv"}}) + + require.NoError(t, c.Set(ctx, "pipe", viewDeps, []byte("v1"), 10*time.Second)) + c.Wait() + val, _, err := c.Get(ctx, "pipe", viewDeps) + require.NoError(t, err) + require.Equal(t, []byte("v1"), val) + + // A write to an unrelated table leaves the view-keyed entry intact. + _, err = c.Invalidate(ctx, []Namespace{{Table: "other"}}) + require.NoError(t, err) + still, _, err := c.Get(ctx, "pipe", viewDeps) + require.NoError(t, err) + require.Equal(t, []byte("v1"), still, "an unrelated write must not cascade") + + // A write to the BASE table cascades to the view and orphans the entry. + _, err = c.Invalidate(ctx, []Namespace{{Table: "page_views"}}) + require.NoError(t, err) + after, _, err := c.Get(ctx, "pipe", viewDeps) + assert.NoError(t, err) + assert.Nil(t, after, "a base-table write must cascade-invalidate the view-keyed entry") +} + // Invalidate with an empty-scope namespace bumps the whole table, which must // orphan that table's scoped entries too — not just the whole-table view. func TestLocalCache_Invalidate_WholeTable(t *testing.T) { diff --git a/internal/cache/version_manager.go b/internal/cache/version_manager.go index c76b2b3c..6caeee7f 100644 --- a/internal/cache/version_manager.go +++ b/internal/cache/version_manager.go @@ -17,6 +17,7 @@ type VersionManager struct { tableVersions map[string]uint64 // -> table_version namespaceVersions map[string]uint64 //
.. -> namespace_version + dependents map[string][]string conn *nats.Conn } @@ -27,10 +28,33 @@ func NewVersionManager(conn *nats.Conn) *VersionManager { return &VersionManager{ tableVersions: make(map[string]uint64), namespaceVersions: make(map[string]uint64), + dependents: make(map[string][]string), conn: conn, } } +// SetDependents installs the base-table -> dependent-view cascade (NATS-encoded on +// both sides), replacing any prior one. Called on each schema refresh. A nil map +// clears the cascade. The slice values are retained as-is; the caller must not +// mutate them afterward (the registry hands over a fresh map each refresh). +func (vm *VersionManager) SetDependents(dependents map[string][]string) { + vm.mu.Lock() + defer vm.mu.Unlock() + if dependents == nil { + dependents = make(map[string][]string) + } + vm.dependents = dependents +} + +// DependentsOf returns the view namespaces a write to table must also bump. Nil +// when table feeds no views. Read under the lock; the returned slice is owned by +// the manager and must not be mutated. +func (vm *VersionManager) DependentsOf(table string) []string { + vm.mu.RLock() + defer vm.mu.RUnlock() + return vm.dependents[table] +} + type Namespace struct { Table string Scope string diff --git a/internal/chsql/chsql.go b/internal/chsql/chsql.go index e91ba8b5..f1cce3df 100644 --- a/internal/chsql/chsql.go +++ b/internal/chsql/chsql.go @@ -41,3 +41,17 @@ func QuoteIdent(name string) string { func BindUnsafe(name string) bool { return strings.ContainsRune(name, '?') } + +// stringEscaper escapes a ClickHouse string-literal body: a backslash becomes +// `\\` and a single quote becomes `”` (ClickHouse accepts the doubled quote). A +// single left-to-right pass, so neither replacement re-processes the other's +// output. +var stringEscaper = strings.NewReplacer(`\`, `\\`, `'`, `''`) + +// QuoteString renders a value as a single-quoted ClickHouse string literal, safe +// to inline into SQL text. It is the value twin of QuoteIdent, for the paths that +// must emit a literal rather than bind a positional `?` — notably pipe parameter +// substitution, which inlines values into the SQL string (see pipes.BindParams). +func QuoteString(s string) string { + return "'" + stringEscaper.Replace(s) + "'" +} diff --git a/internal/query/ident.go b/internal/chsql/nats.go similarity index 76% rename from internal/query/ident.go rename to internal/chsql/nats.go index 7fcba6ca..fc2a179e 100644 --- a/internal/query/ident.go +++ b/internal/chsql/nats.go @@ -1,4 +1,4 @@ -package query +package chsql import ( "bytes" @@ -8,6 +8,10 @@ import ( // SafeEncodeNATS converts any ClickHouse table name into a safe, single NATS token. // It preserves alphanumerics and underscores, but percent-encodes everything else. +// It is the single encoder shared by every side of cache invalidation — the ingest +// worker (write side), the pipe/structured-query handlers (read side), and the +// schema registry's dependency cascade — so all three build identical namespace +// keys for the same table. func SafeEncodeNATS(raw string) string { var buf bytes.Buffer for i := 0; i < len(raw); i++ { diff --git a/internal/query/ident_test.go b/internal/chsql/nats_test.go similarity index 99% rename from internal/query/ident_test.go rename to internal/chsql/nats_test.go index 24e6cb34..9c530bbe 100644 --- a/internal/query/ident_test.go +++ b/internal/chsql/nats_test.go @@ -1,4 +1,4 @@ -package query +package chsql import ( "testing" diff --git a/internal/chsql/quote_test.go b/internal/chsql/quote_test.go new file mode 100644 index 00000000..290e3618 --- /dev/null +++ b/internal/chsql/quote_test.go @@ -0,0 +1,17 @@ +package chsql + +import "testing" + +func TestQuoteString(t *testing.T) { + tests := []struct{ in, want string }{ + {"acme", "'acme'"}, + {"a'b", "'a''b'"}, + {"a\\b", "'a\\\\b'"}, + {"org_id = '1'", "'org_id = ''1'''"}, + } + for _, tt := range tests { + if got := QuoteString(tt.in); got != tt.want { + t.Errorf("QuoteString(%q) = %q, want %q", tt.in, got, tt.want) + } + } +} diff --git a/internal/discovery/deps_test.go b/internal/discovery/deps_test.go new file mode 100644 index 00000000..4d26d808 --- /dev/null +++ b/internal/discovery/deps_test.go @@ -0,0 +1,369 @@ +package discovery + +import ( + "context" + "io" + "log/slog" + "strings" + "testing" + + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" +) + +func tbl(name string, cols ...string) *TableSchema { + ts := &TableSchema{Name: name} + for _, c := range cols { + ts.Columns = append(ts.Columns, Column{Name: c, Type: "UInt64"}) + } + return ts +} + +// IsKnown reports whether a first-level name is safe to fold directly: a base table +// or a view that flattens cleanly to base tables. An unfoldable view (a source it +// can't resolve) and an unknown name are not — the caller TTL-floors those. +func TestIsKnown(t *testing.T) { + tables := []*TableSchema{tbl("base_a", "x"), tbl("base_b", "x"), tbl("v_norm", "x"), tbl("mv1", "x"), tbl("mv2", "x"), tbl("mv_bad", "x")} + viewSources := map[string][]string{ + "v_norm": {"base_a"}, // normal view over a base table + "mv1": {"base_a", "base_b"}, // MV over two sources + "mv2": {"mv1"}, // chained: mv2 -> mv1 -> {base_a, base_b} + "mv_bad": {"ghost"}, // a view over an UNDISCOVERED source: unfoldable + } + sr := NewSchemaRegistryForTest(tables, viewSources) + + known := []string{"base_a", "base_b", "v_norm", "mv1", "mv2"} + for _, n := range known { + assert.True(t, sr.IsKnown(n), "%s should be foldable", n) + } + notKnown := []string{"mv_bad", "ghost", "who_dis"} + for _, n := range notKnown { + assert.False(t, sr.IsKnown(n), "%s must not be foldable", n) + } +} + +// A view we KNOW is a view (engine = View) but whose definition did not parse — +// present in `views`, absent from mvSources — must be reported not-known, never +// mistaken for a base table (which would fold a version nothing maintains). It sits +// in `tables` too (system.columns lists view columns), which is the trap. +func TestIsKnown_UnparsedViewNotFoldable(t *testing.T) { + sr := NewSchemaRegistryForTest([]*TableSchema{tbl("weird_view", "x"), tbl("base", "x")}, map[string][]string{}) + sr.tables["weird_view"].isView = true // a view with no mapped edges (unparsed definition) + + assert.False(t, sr.IsKnown("weird_view"), "an unparsed view must not be foldable") + assert.True(t, sr.IsKnown("base"), "a real base table is foldable") +} + +// computeChangedViews evicts a view redefined in place AND the downstream closure +// of views that transitively read it — staleness no base-table write would catch. +func TestComputeChangedViews(t *testing.T) { + enc := chsql.SafeEncodeNATS + view := func(asSelect string, sources ...string) *tableInfo { + return &tableInfo{isView: true, asSelect: asSelect, sources: sources} + } + base := &tableInfo{schema: &TableSchema{}} + + tests := []struct { + name string + old, new map[string]*tableInfo + want []string // pre-encode names, in sorted order + }{ + { + name: "no change yields nothing", + old: map[string]*tableInfo{"base": base, "v": view("SELECT * FROM base", "base")}, + new: map[string]*tableInfo{"base": base, "v": view("SELECT * FROM base", "base")}, + want: nil, + }, + { + name: "direct redefinition only", + old: map[string]*tableInfo{"base": base, "v": view("SELECT a FROM base", "base")}, + new: map[string]*tableInfo{"base": base, "v": view("SELECT b FROM base", "base")}, + want: []string{"v"}, + }, + { + name: "redefining the middle view also evicts the top (the #1 fix)", + old: map[string]*tableInfo{ + "base": base, + "v_mid": view("SELECT a FROM base", "base"), "v_top": view("SELECT * FROM v_mid", "v_mid"), + }, + new: map[string]*tableInfo{ + "base": base, + "v_mid": view("SELECT b FROM base", "base"), "v_top": view("SELECT * FROM v_mid", "v_mid"), + }, + want: []string{"v_mid", "v_top"}, + }, + { + name: "deep chain fans all the way up", + old: map[string]*tableInfo{ + "base": base, + "v1": view("X", "base"), "v2": view("q2", "v1"), "v3": view("q3", "v2"), + }, + new: map[string]*tableInfo{ + "base": base, + "v1": view("Y", "base"), "v2": view("q2", "v1"), "v3": view("q3", "v2"), + }, + want: []string{"v1", "v2", "v3"}, + }, + { + name: "a view cycle terminates", + old: map[string]*tableInfo{"a": view("A", "b"), "b": view("B", "a")}, + new: map[string]*tableInfo{"a": view("A2", "b"), "b": view("B", "a")}, + want: []string{"a", "b"}, + }, + { + name: "a brand-new downstream reader is still evicted (harmless)", + old: map[string]*tableInfo{"base": base, "v_mid": view("SELECT a FROM base", "base")}, + new: map[string]*tableInfo{ + "base": base, "v_mid": view("SELECT b FROM base", "base"), + "v_new": view("SELECT * FROM v_mid", "v_mid"), + }, + want: []string{"v_mid", "v_new"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + var want []string + for _, n := range tt.want { + want = append(want, enc(n)) + } + assert.Equal(t, want, computeChangedViews(tt.old, tt.new)) + }) + } +} + +// Dependents is the write-side cascade: base table -> the views a write to it must +// also evict, transitively, both NATS-encoded. It is the precomputed reverse of the +// view->source edges; an unfoldable view contributes no entry. +func TestDependents(t *testing.T) { + enc := chsql.SafeEncodeNATS + tests := []struct { + name string + tables []*TableSchema + viewSources map[string][]string + want map[string][]string // base -> dependent views (pre-encode) + }{ + { + name: "single view over a base", + tables: []*TableSchema{tbl("base", "x"), tbl("v", "x")}, + viewSources: map[string][]string{"v": {"base"}}, + want: map[string][]string{"base": {"v"}}, + }, + { + name: "chained views both cascade off the base", + tables: []*TableSchema{tbl("base", "x"), tbl("mv1", "x"), tbl("mv2", "x")}, + viewSources: map[string][]string{"mv1": {"base"}, "mv2": {"mv1"}}, + want: map[string][]string{"base": {"mv1", "mv2"}}, + }, + { + name: "view over two sources cascades off each", + tables: []*TableSchema{tbl("a", "x"), tbl("b", "x"), tbl("v", "x")}, + viewSources: map[string][]string{"v": {"a", "b"}}, + want: map[string][]string{"a": {"v"}, "b": {"v"}}, + }, + { + name: "unfoldable view yields no cascade", + tables: []*TableSchema{tbl("base", "x"), tbl("v", "x")}, + viewSources: map[string][]string{"v": {"ghost"}}, + want: map[string][]string{}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + sr := NewSchemaRegistryForTest(tt.tables, tt.viewSources) + got := sr.Dependents() + want := make(map[string][]string, len(tt.want)) + for base, views := range tt.want { + ev := make([]string, len(views)) + for i, v := range views { + ev[i] = enc(v) + } + want[enc(base)] = ev + } + require.Len(t, got, len(want)) + for base, views := range want { + assert.ElementsMatch(t, views, got[base], "cascade for %s", base) + } + }) + } +} + +// mkInfos assembles a tableInfo map the way Refresh would, for contentHash tests: +// base tables carry a schema; a name in sources/defs is marked a view. +func mkInfos(tables map[string]*TableSchema, sources map[string][]string, defs map[string]string) map[string]*tableInfo { + m := make(map[string]*tableInfo) + for n, ts := range tables { + m[n] = &tableInfo{schema: ts} + } + mark := func(n string) *tableInfo { + t := m[n] + if t == nil { + t = &tableInfo{} + m[n] = t + } + t.isView = true + return t + } + for n, s := range sources { + mark(n).sources = s + } + for n, d := range defs { + mark(n).asSelect = d + } + return m +} + +func TestContentHash_DeterministicAndSensitive(t *testing.T) { + tables := map[string]*TableSchema{ + "a": tbl("a", "x", "y"), + "b": tbl("b", "z"), + } + mv := map[string][]string{"v": {"a"}} + defs := map[string]string{"v": "SELECT x, y FROM a"} + + base := contentHash(mkInfos(tables, mv, defs)) + + // Determinism: a re-run over equal content (maps built independently) matches. + assert.Equal(t, base, contentHash(mkInfos( + map[string]*TableSchema{"b": tbl("b", "z"), "a": tbl("a", "x", "y")}, + map[string][]string{"v": {"a"}}, + map[string]string{"v": "SELECT x, y FROM a"}, + )), "hash must be stable regardless of map iteration order") + + // Sensitivity: each input changing flips the hash. + assert.NotEqual(t, base, contentHash(mkInfos(map[string]*TableSchema{"a": tbl("a", "x", "y", "extra"), "b": tbl("b", "z")}, mv, defs)), "a new column must change the hash") + assert.NotEqual(t, base, contentHash(mkInfos(tables, map[string][]string{"v": {"a", "b"}}, defs)), "a new view->source edge must change the hash") + assert.NotEqual(t, base, contentHash(mkInfos(tables, mv, map[string]string{"v": "SELECT x, y FROM a WHERE z > 1"})), "a redefined view body (same sources) must change the hash") +} + +// scriptedConn returns canned rows for the system.columns and system.tables +// queries (distinguished by SQL substring), and can be told to fail either, +// driving Refresh's change-detection and atomic-no-op behavior without ClickHouse. +type scriptedConn struct { + driver.Conn + columnRows [][]any // [table, name, type, default_kind] + tableRows [][]any // [name, engine, as_select, dependencies_table([]string)] + explainSources []string // tables ResolveTables (EXPLAIN QUERY TREE) reports for a view + explainCalls int // EXPLAIN queries issued (pass-2 runs) — proves the no-op skip + failTables bool +} + +func (c *scriptedConn) Query(_ context.Context, query string, _ ...any) (driver.Rows, error) { + switch { + case strings.HasPrefix(query, "EXPLAIN"): + // Stand in for ClickHouse resolving a view's definition: emit one + // "table_name: .
" line per configured source. + c.explainCalls++ + rows := make([][]any, len(c.explainSources)) + for i, t := range c.explainSources { + rows[i] = []any{" TABLE id: 1, table_name: test." + t} + } + return &scriptedRows{rows: rows}, nil + case strings.Contains(query, "system.columns"): + return &scriptedRows{rows: c.columnRows}, nil + case strings.Contains(query, "system.tables"): + if c.failTables { + return nil, assert.AnError + } + return &scriptedRows{rows: c.tableRows}, nil + } + return &emptyRows{}, nil +} + +type scriptedRows struct { + driver.Rows + rows [][]any + i int +} + +func (r *scriptedRows) Next() bool { return r.i < len(r.rows) } +func (r *scriptedRows) Close() error { return nil } +func (r *scriptedRows) Err() error { return nil } +func (r *scriptedRows) ColumnTypes() []driver.ColumnType { return nil } +func (r *scriptedRows) Scan(dest ...any) error { + row := r.rows[r.i] + r.i++ + for k := range dest { + switch d := dest[k].(type) { + case *string: + d2, _ := row[k].(string) + *d = d2 + case *[]string: + if row[k] == nil { + *d = nil + } else { + *d = row[k].([]string) + } + } + } + return nil +} + +func newScriptedRegistry(conn *scriptedConn) *SchemaRegistry { + return NewSchemaRegistry(conn, "test", 0, slog.New(slog.NewTextHandler(io.Discard, nil))) +} + +// Refresh notifies onRefresh only when the schema content actually changed, hands +// over the up-to-date cascade, and flags a redefined view (same sources, new body) +// in ChangedViews so the cache can evict it directly. +func TestRefresh_NotifiesOnChange(t *testing.T) { + enc := chsql.SafeEncodeNATS + conn := &scriptedConn{ + columnRows: [][]any{{"base", "x", "UInt64", ""}}, + tableRows: [][]any{{"v", "View", "SELECT x FROM base", nil}}, + explainSources: []string{"base"}, + } + sr := newScriptedRegistry(conn) + ctx := context.Background() + + var snaps []DependencySnapshot + sr.SetOnRefresh(func(s DependencySnapshot) { snaps = append(snaps, s) }) + + require.NoError(t, sr.Refresh(ctx)) + require.Len(t, snaps, 1, "first refresh must notify") + assert.ElementsMatch(t, []string{enc("v")}, snaps[0].Cascade[enc("base")], "cascade must map base -> v") + assert.Empty(t, snaps[0].ChangedViews, "no views changed on the first refresh") + explainsAfterFirst := conn.explainCalls + require.Equal(t, 1, explainsAfterFirst, "first refresh resolves the one view via EXPLAIN") + + require.NoError(t, sr.Refresh(ctx)) + require.Len(t, snaps, 1, "an identical refresh must NOT notify") + assert.Equal(t, explainsAfterFirst, conn.explainCalls, "an identical refresh must NOT re-run EXPLAIN — pass 2 is skipped on a no-op") + + // Redefine the view body (same source set) — must be detected and reported. + conn.tableRows = [][]any{{"v", "View", "SELECT x FROM base WHERE x > 1", nil}} + require.NoError(t, sr.Refresh(ctx)) + require.Len(t, snaps, 2, "a view-body redefinition must notify") + assert.Greater(t, conn.explainCalls, explainsAfterFirst, "a redefinition changes the cheap signals, so EXPLAIN re-runs") + assert.Equal(t, []string{enc("v")}, snaps[1].ChangedViews, "the redefined view must be flagged for direct eviction") + + // The view still resolves to its base after the redefinition. + assert.True(t, sr.IsKnown("v")) + assert.ElementsMatch(t, []string{enc("v")}, sr.Dependents()[enc("base")]) +} + +func TestRefresh_AtomicNoOpOnViewDiscoveryError(t *testing.T) { + conn := &scriptedConn{ + columnRows: [][]any{{"base", "x", "UInt64", ""}}, + tableRows: [][]any{{"v", "View", "SELECT x FROM base", nil}}, + explainSources: []string{"base"}, + } + sr := newScriptedRegistry(conn) + ctx := context.Background() + + notifies := 0 + sr.SetOnRefresh(func(DependencySnapshot) { notifies++ }) + + require.NoError(t, sr.Refresh(ctx)) + require.Equal(t, 1, notifies) + + // The tables query now errors; the whole refresh must be a no-op. + conn.failTables = true + require.Error(t, sr.Refresh(ctx)) + assert.Equal(t, 1, notifies, "a failed refresh must not notify") + + // Last-good tree is fully intact: the view still resolves to its base. + assert.True(t, sr.IsKnown("v"), "the previous good tree must survive a transient discovery error") + assert.ElementsMatch(t, []string{chsql.SafeEncodeNATS("v")}, sr.Dependents()[chsql.SafeEncodeNATS("base")]) +} diff --git a/internal/discovery/discovery.go b/internal/discovery/discovery.go index e3b90c5b..952944a9 100644 --- a/internal/discovery/discovery.go +++ b/internal/discovery/discovery.go @@ -3,12 +3,16 @@ package discovery import ( "context" "fmt" + "hash/fnv" "io" "log/slog" + "slices" + "strings" "sync" "time" "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/Wave-RF/WaveHouse/internal/chsql" "go.opentelemetry.io/otel" ) @@ -40,14 +44,72 @@ func (ts *TableSchema) ColumnNames() []string { return names } -// SchemaRegistry discovers and caches ClickHouse table schemas. +// tableInfo is everything the registry knows about ONE name in the database. Base +// tables and views share this one map (system.columns lists a view's columns too, +// so a view carries a schema as well as its view-only fields). One entry replaces +// what used to be five parallel maps. Rebuilt wholesale on every successful Refresh +// and guarded by SchemaRegistry.mu. +type tableInfo struct { + schema *TableSchema // columns in physical order; drives Get/List/ColumnNames + isView bool // engine is a View family — a view, which ingest never writes + sources []string // a view's immediate source tables (nil for a base table) + asSelect string // a view's SELECT text — diffed across refreshes to catch a redefinition + foldable bool // derived: a view that flattens cleanly to known base tables +} + +// SchemaRegistry discovers and caches ClickHouse table schemas, and derives the +// pipe-cache dependency tree from them. type SchemaRegistry struct { conn driver.Conn database string refreshInterval time.Duration logger *slog.Logger mu sync.RWMutex - tables map[string]*TableSchema + // tables maps every name in the configured database — base tables AND views — + // to its schema, view-ness, sources, definition, and derived foldability (see + // tableInfo). The single source of per-name truth; rebuilt and swapped atomically + // on each successful Refresh. Guarded by mu. + tables map[string]*tableInfo + // cascade maps a base table to the dependent views a write to it must ALSO + // invalidate, both NATS-encoded to match the read/write sides. It is the + // precomputed transitive reverse of the view->source edges: the graph walk + // happens once at refresh (buildDeps), never per request. Pushed into the cache + // via onRefresh so Cache.Invalidate can fan a base-table bump out to its views. + // Guarded by mu. + cascade map[string][]string + // onRefresh, if set, is invoked after every CONTENT-CHANGED refresh with the new + // dependency snapshot, so the cache can install the updated cascade and bump any + // redefined views. Set once via SetOnRefresh. Guarded by mu. + onRefresh func(DependencySnapshot) + // metaHash fingerprints the CHEAP schema signals — columns, view-ness, view + // definition text, and reverse dependencies_table edges — read WITHOUT EXPLAIN. + // Each view's EXPLAIN resolution is a deterministic function of these, so when + // metaHash is unchanged (and the last resolve fully succeeded) Refresh skips the + // per-view EXPLAINs entirely and keeps the last-good tree. Guarded by mu. + metaHash uint64 + // contentHash fingerprints the FULL resolved tree (metaHash's inputs plus the + // EXPLAIN-resolved source edges). onRefresh fires only when it changes, so a + // refresh resolving to the same tree neither re-pushes the cascade nor bumps + // anything. hasRefreshed distinguishes the first refresh from a steady-state one. + // Guarded by mu. + contentHash uint64 + // lastResolveOK is false when the previous refresh's EXPLAIN pass had any failure + // (a broken view, or ClickHouse dropping mid-refresh); it forces the next refresh + // to re-run the EXPLAINs even when metaHash is unchanged, so a transient failure + // self-heals instead of freezing missing edges behind the hash. Guarded by mu. + lastResolveOK bool + hasRefreshed bool +} + +// DependencySnapshot is the per-refresh hand-off from the schema registry to the +// cache. Cascade is the full base-table -> dependent-views map (NATS-encoded) to +// install; ChangedViews are the (NATS-encoded) views whose definition changed this +// refresh and must be invalidated directly (a redefinition with the same sources +// changes results but no base-table write would signal it). Both are ready to use +// as-is — the cache neither parses nor re-encodes them. +type DependencySnapshot struct { + Cascade map[string][]string + ChangedViews []string } // NewSchemaRegistry creates a registry that discovers schemas from system.columns. @@ -57,17 +119,222 @@ func NewSchemaRegistry(conn driver.Conn, database string, refreshInterval time.D database: database, refreshInterval: refreshInterval, logger: logger, - tables: make(map[string]*TableSchema), + tables: make(map[string]*tableInfo), } } -// Refresh queries system.columns and rebuilds the in-memory schema cache. +// Refresh re-reads the schema from ClickHouse, recomputes the derived foldable/ +// cascade sets, and atomically swaps the in-memory caches. It runs in two passes so +// the expensive work is gated on change: +// +// - Pass 1 reads the CHEAP signals (system.columns, plus system.tables metadata: +// view-ness, definitions, reverse dependency edges) and fingerprints them +// (metaHash). If nothing moved since the last refresh — and that refresh fully +// resolved — Refresh stops here: no per-view EXPLAINs, no cascade rebuild, no +// onRefresh. This is the steady-state path. +// - Pass 2 runs only on a change: it resolves each view's sources via EXPLAIN (the +// per-view round-trips), rebuilds the cascade, swaps the tree, and — when the +// resolved tree actually differs — notifies onRefresh so the cache installs the +// new cascade and bumps any redefined views. +// +// It is ALL-OR-NOTHING: a read failure aborts with nothing swapped, leaving the +// last-good tree intact. A pass-2 EXPLAIN failure (a broken view, or ClickHouse +// dropping mid-refresh) is recorded (lastResolveOK) so the next refresh re-resolves +// even when the cheap signals are unchanged, rather than freezing missing edges. func (sr *SchemaRegistry) Refresh(ctx context.Context) error { tracer := otel.GetTracerProvider().Tracer("wavehouse-discovery") ctx, span := tracer.Start(ctx, "SchemaRegistry.Refresh") defer span.End() - rows, err := sr.conn.Query(ctx, + infos, err := sr.discoverColumns(ctx) + if err != nil { + return err + } + // Pass 1 (cheap, no EXPLAIN): view-ness, definitions, and reverse dependency + // edges — enough to fingerprint the schema and decide whether anything moved. + viewDefs, err := sr.discoverViewMeta(ctx, infos) + if err != nil { + // Atomic: a metadata-read failure aborts the whole refresh with nothing + // swapped, so the previous good tree stays in effect. Retried next tick. + return err + } + + // Each view's source edges are a deterministic function of the definitions and + // column schema just read, so metaHash captures every input to the EXPLAIN step. + // If it's unchanged AND the last resolve fully succeeded, the EXPLAIN results + // would be identical — skip the per-view round-trips and keep the last-good tree. + metaHash := contentHash(infos) + sr.mu.RLock() + skip := sr.hasRefreshed && metaHash == sr.metaHash && sr.lastResolveOK + sr.mu.RUnlock() + if skip { + sr.logger.Info("schema registry refreshed", "tables", len(infos), "changed", false) + return nil + } + + // Pass 2 (expensive): something changed, or a prior EXPLAIN failed — resolve the + // view source edges via EXPLAIN and rebuild the derived sets. + resolveOK := sr.resolveViewSources(ctx, infos, viewDefs) + treeHash := contentHash(infos) // now folds in the EXPLAIN-resolved edges + cascade := buildDeps(infos) // sets foldable on each view; returns the cascade + sr.mu.Lock() + changed := !sr.hasRefreshed || treeHash != sr.contentHash + // changedViews drives direct eviction of views redefined in place (and the + // downstream views built on them) — staleness no base-table write would catch. + var changedViews []string + if sr.hasRefreshed { + changedViews = computeChangedViews(sr.tables, infos) + } + sr.tables = infos + sr.cascade = cascade + sr.metaHash = metaHash + sr.contentHash = treeHash + sr.lastResolveOK = resolveOK + sr.hasRefreshed = true + onRefresh := sr.onRefresh + sr.mu.Unlock() + + if changed && onRefresh != nil { + // Fire outside the lock: the callback reaches into the cache, and must not + // be able to deadlock against a concurrent reader holding sr.mu.RLock. + onRefresh(DependencySnapshot{Cascade: cascade, ChangedViews: changedViews}) + } + sr.logger.Info("schema registry refreshed", "tables", len(infos), "changed", changed) + return nil +} + +// buildDeps walks the view->source graph ONCE — the only place it is walked, done +// here at refresh so neither read nor write does it per call. It marks each view +// foldable (flattens cleanly to known base tables) directly on its tableInfo, and +// returns the write-side cascade: each base table mapped to the foldable views +// reading it (transitively), NATS-encoded to match the worker (write) and handler +// (read) sides. Only sets the foldable bit on the entries; no lock, no query. +func buildDeps(tables map[string]*tableInfo) map[string][]string { + cascade := make(map[string][]string) + + // flatten returns the base tables `name` transitively reads and whether that + // flatten is COMPLETE (every leaf is a real base table). A view recurses into its + // sources; a base table is itself; a view with no edges (unparsed), an edge-only + // entry with no schema, or an unknown name is incomplete. path is the cycle guard. + var flatten func(name string, path map[string]struct{}) ([]string, bool) + flatten = func(name string, path map[string]struct{}) ([]string, bool) { + if _, cycle := path[name]; cycle { + return nil, false + } + t := tables[name] + if t == nil { + return nil, false // unknown name + } + if t.isView { + if len(t.sources) == 0 { + return nil, false // a view we couldn't map to sources + } + path[name] = struct{}{} + defer delete(path, name) + baseSet := make(map[string]struct{}) + complete := true + for _, s := range t.sources { + sb, sok := flatten(s, path) + if !sok { + complete = false + } + for _, b := range sb { + baseSet[b] = struct{}{} + } + } + bases := make([]string, 0, len(baseSet)) + for b := range baseSet { + bases = append(bases, b) + } + return bases, complete + } + if t.schema != nil { + return []string{name}, true // a real base table ingest writes + } + return nil, false // tracked only as an edge target — no schema, not a base table + } + + for name, t := range tables { + if !t.isView { + continue + } + bases, complete := flatten(name, make(map[string]struct{})) + if !complete { + continue // unfoldable: IsKnown reports it not-known so the caller TTL-floors + } + t.foldable = true + ev := chsql.SafeEncodeNATS(name) + for _, b := range bases { + eb := chsql.SafeEncodeNATS(b) + cascade[eb] = append(cascade[eb], ev) + } + } + for b := range cascade { + slices.Sort(cascade[b]) + cascade[b] = slices.Compact(cascade[b]) + } + return cascade +} + +// computeChangedViews returns the NATS-encoded views to evict after a refresh: the +// views whose definition (as_select) changed in place since oldTables, PLUS the +// downstream closure of views that transitively read them. A redefined view's +// readers produce stale results too, but their own bodies are unchanged and the +// base-keyed cascade can't reach them — so the redefinition must fan out here. +// Returns nil when nothing changed. `changed` doubles as the visited set, so a +// view cycle terminates. A newly-discovered downstream reader is harmless (nothing +// folds its version yet), so the closure needs no present-before filter. +func computeChangedViews(oldTables, newInfos map[string]*tableInfo) []string { + changed := map[string]struct{}{} + for name, t := range newInfos { + if t.asSelect == "" { + continue // only a view has a definition to compare + } + if old := oldTables[name]; old != nil && old.asSelect != "" && old.asSelect != t.asSelect { + changed[name] = struct{}{} + } + } + if len(changed) == 0 { + return nil + } + + readers := map[string][]string{} // name -> views that DIRECTLY read it + for name, t := range newInfos { + if t.isView { + for _, s := range t.sources { + readers[s] = append(readers[s], name) + } + } + } + queue := make([]string, 0, len(changed)) + for n := range changed { + queue = append(queue, n) + } + for len(queue) > 0 { + n := queue[0] + queue = queue[1:] + for _, r := range readers[n] { + if _, seen := changed[r]; seen { + continue + } + changed[r] = struct{}{} + queue = append(queue, r) + } + } + + out := make([]string, 0, len(changed)) + for name := range changed { + out = append(out, chsql.SafeEncodeNATS(name)) + } + slices.Sort(out) + return out +} + +// discoverColumns reads system.columns and builds the per-name map with schemas +// populated (views included — system.columns lists their columns too). +func (sr *SchemaRegistry) discoverColumns(ctx context.Context) (map[string]*tableInfo, error) { + rows, err := sr.conn.Query( + ctx, `SELECT table, name, type, default_kind FROM system.columns WHERE database = ? @@ -76,43 +343,237 @@ func (sr *SchemaRegistry) Refresh(ctx context.Context) error { sr.database, ) if err != nil { - return fmt.Errorf("query system.columns: %w", err) + return nil, fmt.Errorf("query system.columns: %w", err) } defer func() { _ = rows.Close() }() - tables := make(map[string]*TableSchema) + infos := make(map[string]*tableInfo) for rows.Next() { var tableName, colName, colType, defaultKind string if err := rows.Scan(&tableName, &colName, &colType, &defaultKind); err != nil { - return fmt.Errorf("scan column row: %w", err) + return nil, fmt.Errorf("scan column row: %w", err) } - ts, ok := tables[tableName] + t, ok := infos[tableName] if !ok { - ts = &TableSchema{Name: tableName} - tables[tableName] = ts + t = &tableInfo{schema: &TableSchema{Name: tableName}} + infos[tableName] = t } - col := Column{ + t.schema.Columns = append(t.schema.Columns, Column{ Name: colName, Type: colType, IsNullable: isNullable(colType), HasDefault: defaultKind != "", + }) + } + return infos, rows.Err() +} + +// viewDef pairs a view's name with its SELECT text, carried from pass 1 +// (discoverViewMeta) to pass 2 (resolveViewSources) for EXPLAIN resolution. +type viewDef struct{ name, asSelect string } + +// discoverViewMeta is Refresh's PASS 1: it reads the CHEAP view signals from +// system.tables — view-ness, the SELECT definition, and the reverse +// dependencies_table edges (a source -> its attached MV) — and returns the view +// definitions for pass 2 to resolve. No EXPLAIN runs here, so Refresh can fingerprint +// the schema and skip pass 2 entirely when nothing changed. +func (sr *SchemaRegistry) discoverViewMeta(ctx context.Context, infos map[string]*tableInfo) ([]viewDef, error) { + // get returns the entry for name, creating a schema-less one if the name appeared + // only as an edge target (so a view/edge is never silently dropped; a schema-less + // entry is never treated as a base table — see buildDeps/IsKnown). + get := func(name string) *tableInfo { + t := infos[name] + if t == nil { + t = &tableInfo{} + infos[name] = t + } + return t + } + + var toResolve []viewDef + rows, err := sr.conn.Query(ctx, + "SELECT name, engine, as_select, dependencies_table FROM system.tables WHERE database = ?", + sr.database) + if err != nil { + return nil, fmt.Errorf("query system.tables: %w", err) + } + for rows.Next() { + var name, engine, asSelect string + var dependents []string + if err := rows.Scan(&name, &engine, &asSelect, &dependents); err != nil { + _ = rows.Close() + return nil, fmt.Errorf("scan table row: %w", err) + } + t := get(name) + if strings.Contains(engine, "View") { + t.isView = true + } + if asSelect != "" { + t.asSelect = asSelect + toResolve = append(toResolve, viewDef{name, asSelect}) + } + // Reverse edge: this row is a SOURCE table; each dependent is an attached MV. + for _, d := range dependents { + get(d).sources = append(get(d).sources, name) + } + } + rerr := rows.Err() + _ = rows.Close() + if rerr != nil { + return nil, rerr + } + return toResolve, nil +} + +// resolveViewSources is Refresh's PASS 2 — the expensive one, run only when pass 1's +// cheap fingerprint changed. It asks ClickHouse (EXPLAIN QUERY TREE, see +// ResolveTables) for each view's source tables, unions them with the reverse edges +// pass 1 recorded, and de-dups. A view whose definition can't be resolved (it reads a +// table function, a cross-database table) simply gets no forward edge — a pipe reading +// it then over-resolves or goes stale per the documented rules, never a silent wrong +// answer. It returns false if ANY EXPLAIN errored (a broken view, or ClickHouse +// dropping mid-refresh) so Refresh re-resolves next tick instead of trusting a partial +// tree behind an unchanged metaHash. +func (sr *SchemaRegistry) resolveViewSources(ctx context.Context, infos map[string]*tableInfo, toResolve []viewDef) bool { + allResolved := true + for _, v := range toResolve { + srcs, perr := ResolveTables(ctx, sr.conn, sr.database, v.asSelect) + if perr != nil { + sr.logger.Debug("view definition not resolvable via EXPLAIN; relying on dependencies_table", "view", v.name, "error", perr) + allResolved = false + continue + } + infos[v.name].sources = append(infos[v.name].sources, srcs...) + } + + for _, t := range infos { + if len(t.sources) > 1 { + slices.Sort(t.sources) + t.sources = slices.Compact(t.sources) + } + } + return allResolved +} + +// AllBaseTables returns the name of every base table (not a view) — the tables +// ClickHouse writes go to. It is the over-resolve fallback: a pipe whose exact +// tables can't be resolved is treated as depending on all of them, so any write +// evicts it. Coarse, but never a stale read. +func (sr *SchemaRegistry) AllBaseTables() []string { + sr.mu.RLock() + defer sr.mu.RUnlock() + out := make([]string, 0, len(sr.tables)) + for name, t := range sr.tables { + if !t.isView && t.schema != nil { + out = append(out, name) + } + } + return out +} + +// Database returns the configured ClickHouse database this registry tracks. +func (sr *SchemaRegistry) Database() string { return sr.database } + +// contentHash fingerprints everything dependency resolution derives from the schema: +// per name, its columns, view-ness, source edges, and definition. Names are sorted +// so the hash is stable regardless of map iteration order (an unsorted hash would +// flap and re-fire onRefresh every refresh). Conservative by construction — a false +// "changed" only costs a recompute; a real change can never present as "unchanged" +// because every relevant input is folded in. +func contentHash(infos map[string]*tableInfo) uint64 { + h := fnv.New64a() + write := func(s string) { _, _ = h.Write([]byte(s)); _, _ = h.Write([]byte{0}) } + + names := make([]string, 0, len(infos)) + for n := range infos { + names = append(names, n) + } + slices.Sort(names) + for _, n := range names { + t := infos[n] + write("n") + write(n) + if t.schema != nil { + for _, c := range t.schema.Columns { + write(c.Name) + write(c.Type) + if c.HasDefault { + write("d") + } + } + } + if t.isView { + write("v") + } + // Sort/de-dup a copy before hashing so the fingerprint is order-independent: + // pass 1 leaves reverse edges in system.tables row order, which would + // otherwise flap metaHash between refreshes (pass 2's edges are already sorted). + srcs := slices.Clone(t.sources) + slices.Sort(srcs) + for _, s := range slices.Compact(srcs) { + write("s") + write(s) + } + if t.asSelect != "" { + write("q") + write(t.asSelect) } - ts.Columns = append(ts.Columns, col) } + return h.Sum64() +} + +// IsKnown reports whether name is SAFE to fold directly into a cache key — i.e. +// its version is reliably maintained on writes. That is true for a real base +// table (ingest writes it) and for a foldable view (a write to a source bumps it +// via the cascade), but NOT for an unfoldable view (unparsed definition, or a +// cross-database/unknown source) nor an unknown name: the caller treats those as +// unresolved and TTL-floors the result rather than trust an unmaintained version. +// Pure in-memory lookup over the map built during Refresh — no query. +func (sr *SchemaRegistry) IsKnown(name string) bool { + sr.mu.RLock() + defer sr.mu.RUnlock() + t := sr.tables[name] + if t == nil { + return false // unknown name + } + if t.isView { + return t.foldable // a view: safe only if it flattens cleanly to base tables + } + return t.schema != nil // a real base table ingest writes +} +// Dependents returns a copy of the current cascade: each (NATS-encoded) base table +// mapped to the (NATS-encoded) views a write to it must also invalidate. The cache +// installs this so a base-table bump fans out to the views reading it. main pushes +// it through onRefresh; an out-of-band caller (e.g. a test wiring its own cache) +// can pull the current value directly. +func (sr *SchemaRegistry) Dependents() map[string][]string { + sr.mu.RLock() + defer sr.mu.RUnlock() + out := make(map[string][]string, len(sr.cascade)) + for k, v := range sr.cascade { + out[k] = append([]string(nil), v...) + } + return out +} + +// SetOnRefresh registers the callback invoked after every content-changed Refresh +// with the new dependency snapshot. Set once at wiring time, before the periodic +// refresh loop starts, so the cache stays in lock-step with the schema. +func (sr *SchemaRegistry) SetOnRefresh(fn func(DependencySnapshot)) { sr.mu.Lock() - sr.tables = tables + sr.onRefresh = fn sr.mu.Unlock() - sr.logger.Info("schema registry refreshed", "tables", len(tables)) - - return nil } // Get returns the schema for a table, or nil if not found. func (sr *SchemaRegistry) Get(name string) *TableSchema { sr.mu.RLock() defer sr.mu.RUnlock() - return sr.tables[name] + if t := sr.tables[name]; t != nil { + return t.schema + } + return nil } // List returns all discovered table schemas. @@ -120,8 +581,10 @@ func (sr *SchemaRegistry) List() []*TableSchema { sr.mu.RLock() defer sr.mu.RUnlock() result := make([]*TableSchema, 0, len(sr.tables)) - for _, ts := range sr.tables { - result = append(result, ts) + for _, t := range sr.tables { + if t.schema != nil { + result = append(result, t.schema) + } } return result } @@ -203,12 +666,32 @@ func isNullable(chType string) bool { // NewSchemaRegistryFromMap creates a SchemaRegistry pre-loaded with the given // table schemas. Intended for testing — no ClickHouse connection is required. func NewSchemaRegistryFromMap(tables []*TableSchema) *SchemaRegistry { - m := make(map[string]*TableSchema, len(tables)) + m := make(map[string]*tableInfo, len(tables)) for _, t := range tables { - m[t.Name] = t + m[t.Name] = &tableInfo{schema: t} } return &SchemaRegistry{ tables: m, logger: slog.New(slog.NewTextHandler(io.Discard, nil)), } } + +// NewSchemaRegistryForTest creates a registry pre-loaded with table schemas AND +// view->source edges, with the foldable/cascade sets derived from them, for testing +// dependency resolution without a ClickHouse connection. Marked as already +// refreshed, mimicking a once-refreshed registry. +func NewSchemaRegistryForTest(tables []*TableSchema, viewSources map[string][]string) *SchemaRegistry { + sr := NewSchemaRegistryFromMap(tables) + for name, srcs := range viewSources { + t := sr.tables[name] + if t == nil { + t = &tableInfo{} + sr.tables[name] = t + } + t.isView = true + t.sources = srcs + } + sr.cascade = buildDeps(sr.tables) + sr.hasRefreshed = true + return sr +} diff --git a/internal/discovery/discovery_test.go b/internal/discovery/discovery_test.go index 67558be5..0ae9c1ae 100644 --- a/internal/discovery/discovery_test.go +++ b/internal/discovery/discovery_test.go @@ -155,7 +155,9 @@ func TestRetryRefresh_SucceedsOnFirstAttempt(t *testing.T) { // regression (the misbehaviour would sleep `initialBackoff` = 1h). assert.Less(t, time.Since(start), 250*time.Millisecond, "should not have slept") assert.Equal(t, int32(0), atomic.LoadInt32(&attempts), "onAttempt should only fire on failure") - assert.Equal(t, int32(1), conn.calls.Load(), "exactly one Query call expected") + // A successful Refresh makes two queries: system.columns (schema) + system.tables + // (the materialized-view -> source dependency map). + assert.Equal(t, int32(2), conn.calls.Load(), "two Query calls on a successful Refresh") } // TestRetryRefresh_RetriesUntilSuccess verifies the loop keeps trying through @@ -173,7 +175,9 @@ func TestRetryRefresh_RetriesUntilSuccess(t *testing.T) { }) require.NoError(t, err) - assert.Equal(t, int32(3), conn.calls.Load(), "two failures + one success") + // Two failed attempts (1 query each — system.columns errors before the + // dependency query) + the succeeding attempt's two queries (columns + tables). + assert.Equal(t, int32(4), conn.calls.Load(), "two failures + one success") require.Len(t, captured, 2) assert.ErrorIs(t, captured[0], errFirst) assert.ErrorIs(t, captured[1], errSecond) diff --git a/internal/discovery/explain.go b/internal/discovery/explain.go new file mode 100644 index 00000000..e544d1e0 --- /dev/null +++ b/internal/discovery/explain.go @@ -0,0 +1,383 @@ +package discovery + +import ( + "context" + "fmt" + "slices" + "strings" + + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/metric" +) + +// prunedTablesCounter counts table dependencies dropped by dead-branch pruning +// (a parameter-gated UNION arm whose WHERE folded to a constant false). It lets +// operators measure how often the precision optimization fires. A no-op until a +// meter provider is configured, so it is safe to use unconditionally. +var prunedTablesCounter, _ = otel.Meter("wavehouse-pipes").Int64Counter( + "wavehouse_pipe_dep_tables_pruned_total", + metric.WithDescription("Table dependencies dropped by dead-branch (constant-false) pruning during pipe resolution"), +) + +// ResolveTables asks ClickHouse which tables a query reads, by running +// EXPLAIN QUERY TREE and reading the table nodes off its own analysis. This +// replaces static SQL parsing entirely — ClickHouse resolves the query, we just +// read the answer. +// +// EXPLAIN QUERY TREE is deliberately the PRE-optimization form: it keeps a table +// even when the query is answered from metadata (SELECT count() FROM t), and it +// does NOT drop empty tables or trivial counts the way EXPLAIN PLAN / query_log do. +// So it never UNDER-resolves, the invariant we need. +// +// On top of the raw tree we apply two refinements (parseExplainTables): +// +// - joinGet/dictGet — read-bearing functions whose target is a NAME constant +// rather than a TABLE node — are tracked (the dict via its backing table). Both +// are real reads a write must evict. +// - Dead-branch pruning — a parameter-gated UNION arm whose WHERE folded to a +// constant false (e.g. {{source}}='web' makes the 'mobile' arm 'web'='mobile') +// reads nothing, so its tables are dropped. This is data-INDEPENDENT (pure +// constant folding) and so safe, BUT it makes the resolved set depend on the +// bound parameter values — the caller must therefore cache the result per BOUND +// query, not per template, or a different binding would reuse a wrongly-pruned +// set (see api.PipesHandler.resolveDeps). Pruning fails safe toward KEEPING: any +// unrecognized shape leaves the table tracked (over-resolve), never dropped. +// +// It returns the local (configured-database) table/view names. A reference to a +// DIFFERENT database, and a table function (s3()/numbers()/…), produce no trackable +// local table and are silently omitted — those pipes are unsupported and their +// results may go stale (documented). An EXPLAIN error — a write/DDL pipe, a missing +// table, or an unreachable ClickHouse — is returned so the caller can fall back to +// over-resolving (all tables) rather than risk an under-resolution. +func ResolveTables(ctx context.Context, conn driver.Conn, database, sql string) ([]string, error) { + rows, err := conn.Query(ctx, "EXPLAIN QUERY TREE "+sql) + if err != nil { + return nil, fmt.Errorf("explain query tree: %w", err) + } + defer func() { _ = rows.Close() }() + + var lines []string + for rows.Next() { + var ln string + if err := rows.Scan(&ln); err != nil { + return nil, fmt.Errorf("scan explain row: %w", err) + } + lines = append(lines, ln) + } + if err := rows.Err(); err != nil { + return nil, err + } + + tables, dicts, pruned := parseExplainTables(lines, database) + if pruned > 0 { + prunedTablesCounter.Add(ctx, int64(pruned)) + } + // A dictGet target is the dictionary, not a table; map each to its backing + // ClickHouse table (a CLICKHOUSE-sourced dict in the configured db). A + // file/http/executable dict has no local table and is dropped, like a table + // function. Only queried when the pipe actually reads a dictionary. + if len(dicts) > 0 { + srcs, err := resolveDictSources(ctx, conn, database, dicts) + if err != nil { + return nil, err + } + tables = append(tables, srcs...) + } + slices.Sort(tables) + return slices.Compact(tables), nil +} + +// capture tracks whether the previous line armed extraction of the next string +// constant — joinGet/dictGet render their target as the function's first argument, +// a NAME constant, rather than a TABLE node. +const ( + capNone = iota + capJoin // next string constant is a Join-engine TABLE name + capDict // next string constant is a DICTIONARY name +) + +// qscope is one QUERY node in the tree, tracked by indentation. dead is set when +// the node's WHERE folded to a constant false (the arm reads nothing). Scopes are +// retained for the whole parse so a TABLE seen BEFORE its arm's WHERE can still have +// its deadness resolved at the end. +type qscope struct { + indent int + dead bool +} + +// tableOccurrence records one TABLE node plus the QUERY scope stack enclosing it, +// so its deadness (any enclosing arm dead) is evaluated after the full parse. +type tableOccurrence struct { + table string + scopes []int // indices into the scopes slice +} + +// parseExplainTables extracts dependency names from EXPLAIN QUERY TREE output, +// returning (tables, dicts, prunedCount): +// +// - tables: configured-database table/view names from TABLE nodes (minus +// dead-branch arms) plus the first (NAME) argument of joinGet/joinGetOrNull. +// - dicts: configured-database dictionary names referenced via a dictGet-family +// function, for the caller to map to their backing tables. +// - prunedCount: distinct tables dropped because every occurrence was in a +// constant-false arm (for the metric). +// +// Structure (verified against ClickHouse 26.6): a UNION arm is a QUERY node; its +// tables sit under JOIN TREE and its filter under a sibling WHERE. ClickHouse folds +// a parameter-gated predicate, so a dead arm's WHERE child is exactly +// "CONSTANT … UInt64_0" — and ONLY the line immediately after WHERE is inspected, so +// a real predicate ("WHERE → FUNCTION greater") or a literal compare +// ("WHERE → FUNCTION equals(x, 0)") is never mistaken for a dead arm. A table is +// dropped only when EVERY occurrence is under a dead arm; appearing in any live arm +// keeps it. Kept pure (no ClickHouse) so the brittle string handling is unit-tested. +func parseExplainTables(lines []string, database string) (tables, dicts []string, prunedCount int) { + var scopes []qscope + var stack []int // indices into scopes for currently-open QUERY nodes + var occs []tableOccurrence + dictSet := map[string]struct{}{} + joinTables := map[string]struct{}{} // joinGet targets: always kept (safe), never pruned + capture := capNone + afterWhere := false + + for _, raw := range lines { + trimmed := strings.TrimLeft(raw, " ") + if trimmed == "" { + continue + } + indent := len(raw) - len(trimmed) + + // Close any QUERY scopes we have exited (this line is at or shallower than them). + for len(stack) > 0 && scopes[stack[len(stack)-1]].indent >= indent { + stack = stack[:len(stack)-1] + } + + // The single line after a WHERE is its condition root. If that root is the + // folded constant false, this arm (the enclosing QUERY) is dead. Checking only + // this one line is what makes pruning safe: a real predicate or an "x = 0" + // compare renders as a FUNCTION here, not a bare CONSTANT. + if afterWhere { + afterWhere = false + if len(stack) > 0 && strings.Contains(raw, "constant_value: UInt64_0,") { + scopes[stack[len(stack)-1]].dead = true + } + // Fall through: the WHERE condition root may itself be a joinGet/dictGet + // (e.g. WHERE joinGet(...)), which still needs tracking. + } + + switch { + case strings.HasPrefix(trimmed, "QUERY id:"): + scopes = append(scopes, qscope{indent: indent}) + stack = append(stack, len(scopes)-1) + capture = capNone + continue + case trimmed == "WHERE" || strings.HasPrefix(trimmed, "WHERE "): + afterWhere = true + capture = capNone + continue + } + + // TABLE node — record with its enclosing scope stack (deadness resolved later). + // Gate on the node type ("TABLE ...") so a string-literal value that merely + // contains "table_name:" (a projected label, or a bound parameter value) can't + // be mistaken for a table reference. + if strings.HasPrefix(trimmed, "TABLE ") { + if _, after, found := strings.Cut(raw, "table_name: "); found { + // table_name is NOT always the last field on the line: FINAL appends + // ", final: 1" and SAMPLE ", final: 0, sample_size: …". Keep only the + // qualified name up to the first comma, else the modifier text leaks in. + name, _, _ := strings.Cut(after, ",") + if db, table, ok := strings.Cut(strings.TrimSpace(name), "."); ok && db == database { + occs = append(occs, tableOccurrence{table: table, scopes: append([]int(nil), stack...)}) + } + } + capture = capNone + continue + } + + // joinGet/dictGet: arm capture of the next string constant (the target name). + switch { + case strings.Contains(raw, "function_name: joinGet"): + capture = capJoin + continue + case strings.Contains(raw, "function_name: dictGet"), + strings.Contains(raw, "function_name: dictHas"), + strings.Contains(raw, "function_name: dictGetHierarchy"), + strings.Contains(raw, "function_name: dictGetChildren"), + strings.Contains(raw, "function_name: dictGetDescendants"), + strings.Contains(raw, "function_name: dictIsIn"): + capture = capDict + continue + } + if capture != capNone { + if val, isStr, found := constantString(raw); found { + if isStr { + if db, name, ok := strings.Cut(val, "."); ok { + if db == database { + addCaptured(capture, name, joinTables, dictSet) + } + } else { + addCaptured(capture, val, joinTables, dictSet) // bare name -> configured db + } + } + capture = capNone + } + } + } + + // Resolve table deadness: a table is kept if it has at least one occurrence with + // no dead enclosing arm. A table dropped here (all occurrences dead) is counted. + tableSet := map[string]struct{}{} + seen := map[string]struct{}{} + for _, o := range occs { + seen[o.table] = struct{}{} + dead := false + for _, si := range o.scopes { + if scopes[si].dead { + dead = true + break + } + } + if !dead { + tableSet[o.table] = struct{}{} + } + } + for t := range seen { + if _, kept := tableSet[t]; !kept { + prunedCount++ + } + } + for t := range joinTables { + tableSet[t] = struct{}{} + } + return setSlice(tableSet), setSlice(dictSet), prunedCount +} + +// addCaptured routes a captured NAME to the table or dict set per the armed kind. +func addCaptured(kind int, name string, tables, dicts map[string]struct{}) { + switch kind { + case capJoin: + tables[name] = struct{}{} + case capDict: + dicts[name] = struct{}{} + } +} + +// constantString parses a "constant_value: " line. found reports the line is a +// constant node; isStr reports the value is a quoted String (the only kind that can +// be a table/dict name) and val is its unquoted content. A non-string constant +// (e.g. "UInt64_1") returns isStr=false so the caller stops capturing. +func constantString(ln string) (val string, isStr, found bool) { + _, after, ok := strings.Cut(ln, "constant_value: ") + if !ok { + return "", false, false + } + after = strings.TrimSpace(after) + if !strings.HasPrefix(after, "'") { + return "", false, true // e.g. UInt64_0 — a non-name constant + } + if i := strings.IndexByte(after[1:], '\''); i >= 0 { + return after[1 : 1+i], true, true + } + return "", false, true // unterminated quote — malformed, ignore +} + +// resolveDictSources maps the named dictionaries to their backing ClickHouse +// tables. system.dictionaries.source renders a CLICKHOUSE source as +// "ClickHouse: .
"; only a source table in the configured database is +// trackable (a write to it evicts). Other source kinds (file/http/executable) and +// cross-database sources yield no local table, like a table function. +// +// A NOT_LOADED dictionary reports an EMPTY source — under ClickHouse's default lazy +// loading a dict stays unloaded until first used, so resolving a dictGet pipe before +// the dict's first load would silently drop its backing table (→ stale reads on +// writes to it). So any referenced-but-unloaded dict is force-loaded once, then +// re-read. Best-effort: a dict that can't load is left untracked rather than failing +// the whole resolution. +func resolveDictSources(ctx context.Context, conn driver.Conn, database string, dicts []string) ([]string, error) { + want := make(map[string]struct{}, len(dicts)) + for _, d := range dicts { + want[d] = struct{}{} + } + + readSources := func() (map[string]string, error) { + rows, err := conn.Query(ctx, "SELECT name, source FROM system.dictionaries WHERE database = ?", database) + if err != nil { + return nil, fmt.Errorf("query system.dictionaries: %w", err) + } + defer func() { _ = rows.Close() }() + out := make(map[string]string, len(want)) + for rows.Next() { + var name, source string + if err := rows.Scan(&name, &source); err != nil { + return nil, fmt.Errorf("scan dictionary row: %w", err) + } + if _, ok := want[name]; ok { + out[name] = source + } + } + return out, rows.Err() + } + + sources, err := readSources() + if err != nil { + return nil, err + } + + loaded := false + for name, source := range sources { + if source != "" { + continue // already loaded + } + stmt := "SYSTEM RELOAD DICTIONARY " + chsql.QuoteIdent(database) + "." + chsql.QuoteIdent(name) + if err := conn.Exec(ctx, stmt); err == nil { + loaded = true + } + } + if loaded { + if sources, err = readSources(); err != nil { + return nil, err + } + } + + var out []string + for _, source := range sources { + if table, ok := parseDictSourceTable(source, database); ok { + out = append(out, table) + } + } + return out, nil +} + +// parseDictSourceTable extracts the backing table from a system.dictionaries +// source string, returning ok=false for a non-CLICKHOUSE source or a source in a +// different database (neither is trackable here). +func parseDictSourceTable(source, database string) (string, bool) { + const prefix = "ClickHouse: " + if !strings.HasPrefix(source, prefix) { + return "", false + } + qualified := strings.TrimSpace(strings.TrimPrefix(source, prefix)) + db, table, ok := strings.Cut(qualified, ".") + if !ok { + return qualified, true // bare table name -> configured db + } + if db != database { + return "", false + } + return table, true +} + +// setSlice returns the set's keys as a sorted slice (nil when empty). +func setSlice(set map[string]struct{}) []string { + if len(set) == 0 { + return nil + } + out := make([]string, 0, len(set)) + for s := range set { + out = append(out, s) + } + slices.Sort(out) + return out +} diff --git a/internal/discovery/explain_test.go b/internal/discovery/explain_test.go new file mode 100644 index 00000000..d636bcde --- /dev/null +++ b/internal/discovery/explain_test.go @@ -0,0 +1,316 @@ +package discovery + +import ( + "reflect" + "testing" +) + +// parseExplainTables is the brittle string-handling half of ResolveTables; the +// sample lines below are real EXPLAIN QUERY TREE output shapes (verified against +// ClickHouse 26.6). The ResolveTables round-trip itself is covered by the +// integration test against a live server. +func TestParseExplainTables(t *testing.T) { + tests := []struct { + name string + lines []string + db string + wantTables []string + wantDicts []string + }{ + { + name: "simple join, both local", + lines: []string{ + " TABLE id: 3, alias: __table1, table_name: default.base", + " TABLE id: 5, alias: __table2, table_name: default.other", + }, + db: "default", + wantTables: []string{"base", "other"}, + }, + { + name: "count() keeps the table (the whole point)", + lines: []string{" TABLE id: 3, alias: __table1, table_name: default.events"}, + db: "default", + wantTables: []string{"events"}, + }, + { + name: "weird name with dots+spaces: split on FIRST dot only", + lines: []string{ + " TABLE id: 3, alias: __table1, table_name: default.weird.name 2024", + }, + db: "default", + wantTables: []string{"weird.name 2024"}, + }, + { + name: "foreign database is dropped (unsupported, documented)", + lines: []string{ + " TABLE id: 3, alias: __table1, table_name: otherdb.events", + " TABLE id: 5, alias: __table2, table_name: default.local", + }, + db: "default", + wantTables: []string{"local"}, + }, + { + name: "dedup repeated table (self-join)", + lines: []string{ + " TABLE id: 3, table_name: default.t", + " TABLE id: 9, table_name: default.t", + }, + db: "default", + wantTables: []string{"t"}, + }, + { + name: "table function: no table_name line → nothing", + lines: []string{" QUERY id: 0", " JOIN TREE", " TABLE_FUNCTION numbers"}, + db: "default", + }, + { + name: "no tables (SELECT 1)", + lines: []string{"QUERY id: 0", " PROJECTION COLUMNS", " 1 UInt8"}, + db: "default", + }, + { + name: "joinGet: Join-engine table tracked from the first constant", + lines: []string{ + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'default.jt', constant_value_type: String", + " CONSTANT id: 7, constant_value: 'v', constant_value_type: String", + }, + db: "default", + wantTables: []string{"jt"}, + }, + { + name: "joinGet alongside a FROM table: both tracked", + lines: []string{ + " TABLE id: 3, alias: __table1, table_name: default.users", + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'default.jt', constant_value_type: String", + }, + db: "default", + wantTables: []string{"jt", "users"}, + }, + { + name: "joinGet bare (unqualified) name → configured db", + lines: []string{ + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'jt', constant_value_type: String", + }, + db: "default", + wantTables: []string{"jt"}, + }, + { + name: "joinGet cross-db dropped", + lines: []string{ + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'otherdb.jt', constant_value_type: String", + }, + db: "default", + }, + { + name: "joinGet with a non-string first constant disarms (no false capture)", + lines: []string{ + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: UInt64_1, constant_value_type: UInt64", + " CONSTANT id: 7, constant_value: 'v', constant_value_type: String", + }, + db: "default", + }, + { + name: "dictGet: dictionary captured; the projection's stray '' before it is ignored", + lines: []string{ + " CONSTANT id: 2, constant_value: '', constant_value_type: String", + " FUNCTION id: 3, function_name: dictGet, function_type: ordinary, result_type: String", + " CONSTANT id: 5, constant_value: 'default.mydict', constant_value_type: String", + " CONSTANT id: 6, constant_value: 'val', constant_value_type: String", + }, + db: "default", + wantDicts: []string{"mydict"}, + }, + { + name: "dictGet cross-db dropped", + lines: []string{ + " FUNCTION id: 3, function_name: dictGet, function_type: ordinary, result_type: String", + " CONSTANT id: 5, constant_value: 'otherdb.mydict', constant_value_type: String", + }, + db: "default", + }, + { + name: "joinGet and dictGet together", + lines: []string{ + " TABLE id: 3, table_name: default.users", + " FUNCTION id: 4, function_name: joinGet, function_type: ordinary, result_type: String", + " CONSTANT id: 6, constant_value: 'default.jt', constant_value_type: String", + " FUNCTION id: 8, function_name: dictGetOrDefault, function_type: ordinary, result_type: String", + " CONSTANT id: 9, constant_value: 'default.mydict', constant_value_type: String", + }, + db: "default", + wantTables: []string{"jt", "users"}, + wantDicts: []string{"mydict"}, + }, + { + name: "FINAL: trailing ', final: 1' must not leak into the table name", + lines: []string{" TABLE id: 3, alias: __table1, table_name: default.rmt, final: 1"}, + db: "default", + wantTables: []string{"rmt"}, + }, + { + name: "SAMPLE: trailing ', final: 0, sample_size: …' stripped", + lines: []string{" TABLE id: 3, table_name: default.smp, final: 0, sample_size: 5 / 10"}, + db: "default", + wantTables: []string{"smp"}, + }, + { + name: "string literal containing 'table_name:' is not mistaken for a table", + lines: []string{ + " CONSTANT id: 2, constant_value: 'table_name: default.injected', constant_value_type: String", + " TABLE id: 3, alias: __table1, table_name: default.real", + }, + db: "default", + wantTables: []string{"real"}, + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + gotTables, gotDicts, _ := parseExplainTables(tt.lines, tt.db) + if !reflect.DeepEqual(gotTables, tt.wantTables) { + t.Errorf("tables = %v, want %v", gotTables, tt.wantTables) + } + if !reflect.DeepEqual(gotDicts, tt.wantDicts) { + t.Errorf("dicts = %v, want %v", gotDicts, tt.wantDicts) + } + }) + } +} + +// TestParseExplainTables_DeadBranchPruning covers the parameter-gated UNION case: +// after binding, a dead arm's WHERE folds to "CONSTANT UInt64_0" and its tables are +// dropped, while a live arm (WHERE → 1, a real predicate, or an "x = 0" compare) is +// kept. The fixtures reproduce real EXPLAIN QUERY TREE indentation (ClickHouse 26.6), +// since pruning is structure-aware. The cardinal safety property: pruning fails +// toward KEEPING — only an exact constant-false WHERE child drops a table. +func TestParseExplainTables_DeadBranchPruning(t *testing.T) { + // arm builds one UNION-arm QUERY: a table under JOIN TREE and a WHERE whose + // folded child is the given constant ("UInt64_1" live, "UInt64_0" dead) or a raw + // node line (e.g. a FUNCTION) for a non-folded predicate. + arm := func(table, whereChild string) []string { + return []string{ + " QUERY id: 6, alias: __table2", + " JOIN TREE", + " TABLE id: 9, alias: __table3, table_name: default." + table, + " WHERE", + " " + whereChild, + } + } + wrap := func(arms ...[]string) []string { + out := []string{ + "QUERY id: 0", + " JOIN TREE", + " UNION id: 3, alias: __table1, is_subquery: 1, union_mode: UNION_ALL", + " LIST id: 5, nodes: 2", + } + for _, a := range arms { + out = append(out, a...) + } + return out + } + const ( + live = "CONSTANT id: 10, constant_value: UInt64_1, constant_value_type: UInt8" + dead = "CONSTANT id: 19, constant_value: UInt64_0, constant_value_type: UInt8" + pred = "FUNCTION id: 4, function_name: greater, function_type: ordinary, result_type: UInt8" + eq0 = "FUNCTION id: 4, function_name: equals, function_type: ordinary, result_type: UInt8" + ) + + tests := []struct { + name string + lines []string + wantTables []string + wantPruned int + }{ + { + name: "source='web': mobile arm dead, pruned", + lines: wrap(arm("web_events", live), arm("mobile_events", dead)), + wantTables: []string{"web_events"}, + wantPruned: 1, + }, + { + name: "both arms live: nothing pruned", + lines: wrap(arm("web_events", live), arm("mobile_events", live)), + wantTables: []string{"mobile_events", "web_events"}, + wantPruned: 0, + }, + { + name: "real predicate (WHERE col>5) is NOT a dead arm", + lines: wrap(arm("web_events", pred), arm("mobile_events", dead)), + wantTables: []string{"web_events"}, + wantPruned: 1, + }, + { + name: "WHERE x=0 (equals function, not a bare constant) is NOT dead", + lines: wrap(arm("web_events", eq0), arm("mobile_events", dead)), + wantTables: []string{"web_events"}, + wantPruned: 1, + }, + { + name: "a table in BOTH a live and a dead arm stays tracked", + lines: wrap( + arm("shared", live), + arm("shared", dead), + arm("mobile_events", dead), + ), + wantTables: []string{"shared"}, + wantPruned: 1, // only mobile_events + }, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + gotTables, _, gotPruned := parseExplainTables(tt.lines, "default") + if !reflect.DeepEqual(gotTables, tt.wantTables) { + t.Errorf("tables = %v, want %v", gotTables, tt.wantTables) + } + if gotPruned != tt.wantPruned { + t.Errorf("pruned = %d, want %d", gotPruned, tt.wantPruned) + } + }) + } +} + +func TestParseDictSourceTable(t *testing.T) { + tests := []struct { + name, source, db, want string + ok bool + }{ + {"clickhouse qualified, same db", "ClickHouse: default.dsrc", "default", "dsrc", true}, + {"clickhouse bare name", "ClickHouse: dsrc", "default", "dsrc", true}, + {"clickhouse other db dropped", "ClickHouse: otherdb.dsrc", "default", "", false}, + {"file source untracked", "File: /var/lib/x.csv", "default", "", false}, + {"executable source untracked", "Executable: ./gen.sh", "default", "", false}, + {"http source untracked", "HTTP: http://x/y", "default", "", false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + got, ok := parseDictSourceTable(tt.source, tt.db) + if got != tt.want || ok != tt.ok { + t.Errorf("parseDictSourceTable(%q) = (%q, %v), want (%q, %v)", tt.source, got, ok, tt.want, tt.ok) + } + }) + } +} + +func TestConstantString(t *testing.T) { + tests := []struct { + name, line, wantVal string + wantStr, wantFound bool + }{ + {"quoted string", " CONSTANT id: 6, constant_value: 'default.jt', constant_value_type: String", "default.jt", true, true}, + {"empty string", " CONSTANT id: 2, constant_value: '', constant_value_type: String", "", true, true}, + {"non-string constant", " CONSTANT id: 7, constant_value: UInt64_0, constant_value_type: UInt8", "", false, true}, + {"not a constant line", " TABLE id: 3, table_name: default.users", "", false, false}, + } + for _, tt := range tests { + t.Run(tt.name, func(t *testing.T) { + val, isStr, found := constantString(tt.line) + if val != tt.wantVal || isStr != tt.wantStr || found != tt.wantFound { + t.Errorf("constantString(%q) = (%q,%v,%v), want (%q,%v,%v)", tt.line, val, isStr, found, tt.wantVal, tt.wantStr, tt.wantFound) + } + }) + } +} diff --git a/internal/discovery/fuzz_deps_test.go b/internal/discovery/fuzz_deps_test.go new file mode 100644 index 00000000..ee11a59d --- /dev/null +++ b/internal/discovery/fuzz_deps_test.go @@ -0,0 +1,356 @@ +//go:build fuzzdeps + +// Differential fuzzer for ResolveTables. Excluded from the normal build by the +// fuzzdeps tag; run on demand against a local ClickHouse: +// +// go test -tags fuzzdeps -run TestFuzzDeps ./internal/discovery/ -v +// +// It generates hundreds of diverse queries (nested/UNION/parameter-gated, plus +// targeted joinGet/dictGet/cross-db/FINAL/SAMPLE/injection cases) with a +// construction-based ground truth — it builds each query, so it knows which tables +// sit in live vs. dead arms — then runs the real ResolveTables and flags +// under-resolution (stale-cache BUG, fails the test), over-resolution (pruning +// miss / parser hallucination), and EXPLAIN errors. Creates and drops its own +// fuzz_deps / fuzz_other databases; skips if no ClickHouse is reachable on :9000. +package discovery + +import ( + "context" + "fmt" + "math/rand" + "sort" + "strings" + "testing" + "time" + + "github.com/ClickHouse/clickhouse-go/v2" + "github.com/ClickHouse/clickhouse-go/v2/lib/driver" +) + +const ( + fdb = "fuzz_deps" + fother = "fuzz_other" +) + +var baseShort = []string{"bt0", "bt1", "bt2", "bt3", "bt4", "bt5", "bt6", "bt7"} + +func full(short string) string { return fdb + "." + short } + +type occ struct { + table string + dead bool +} + +func connectFuzz(t *testing.T) driver.Conn { + open := func(db string) (driver.Conn, error) { + return clickhouse.Open(&clickhouse.Options{ + Addr: []string{"127.0.0.1:9000"}, + Auth: clickhouse.Auth{Database: db, Username: "default", Password: ""}, + DialTimeout: 5 * time.Second, + }) + } + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + boot, err := open("default") + if err != nil { + t.Fatalf("open default: %v", err) + } + if err := boot.Ping(ctx); err != nil { + t.Skipf("clickhouse not reachable: %v", err) + } + _ = boot.Exec(ctx, "CREATE DATABASE IF NOT EXISTS "+fdb) + _ = boot.Exec(ctx, "CREATE DATABASE IF NOT EXISTS "+fother) + _ = boot.Close() + conn, err := open(fdb) + if err != nil { + t.Fatalf("open %s: %v", fdb, err) + } + if err := conn.Ping(ctx); err != nil { + t.Fatalf("ping %s: %v", fdb, err) + } + return conn +} + +func setupSchema(t *testing.T, conn driver.Conn) { + ctx := context.Background() + exec := func(s string) { + if err := conn.Exec(ctx, s); err != nil { + t.Fatalf("setup %q: %v", s, err) + } + } + exec("CREATE DATABASE IF NOT EXISTS " + fdb) + exec("CREATE DATABASE IF NOT EXISTS " + fother) + for _, s := range baseShort { + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.%s (x Int32, s String) ENGINE=MergeTree ORDER BY x", fdb, s)) + exec(fmt.Sprintf("INSERT INTO %s.%s SELECT number, toString(number) FROM numbers(5)", fdb, s)) + } + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.`weird.name` (x Int32, s String) ENGINE=MergeTree ORDER BY x", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.`weird.name` SELECT number, toString(number) FROM numbers(5)", fdb)) + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.jt (k String, v String) ENGINE=Join(ANY, LEFT, k)", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.jt VALUES ('a','1')", fdb)) + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.dsrc (id UInt64, val String) ENGINE=MergeTree ORDER BY id", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.dsrc VALUES (1,'one')", fdb)) + exec(fmt.Sprintf("CREATE DICTIONARY IF NOT EXISTS %s.mydict (id UInt64, val String) PRIMARY KEY id SOURCE(CLICKHOUSE(DB '%s' TABLE 'dsrc')) LAYOUT(HASHED()) LIFETIME(0)", fdb, fdb)) + // Deliberately NOT pre-loading mydict, so the dictGet cases exercise + // resolveDictSources' load-before path against a NOT_LOADED dictionary. + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.rmt (x Int32, s String) ENGINE=ReplacingMergeTree ORDER BY x", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.rmt SELECT number, toString(number) FROM numbers(5)", fdb)) + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.smp (x Int32, h UInt32) ENGINE=MergeTree ORDER BY h SAMPLE BY h", fdb)) + exec(fmt.Sprintf("INSERT INTO %s.smp SELECT number, toUInt32(number) FROM numbers(5)", fdb)) + exec(fmt.Sprintf("CREATE TABLE IF NOT EXISTS %s.ext (x Int32, s String) ENGINE=MergeTree ORDER BY x", fother)) + exec(fmt.Sprintf("INSERT INTO %s.ext SELECT number, toString(number) FROM numbers(5)", fother)) +} + +func teardown(conn driver.Conn) { + ctx := context.Background() + _ = conn.Exec(ctx, "DROP DATABASE IF EXISTS "+fdb) + _ = conn.Exec(ctx, "DROP DATABASE IF EXISTS "+fother) +} + +// genGate returns a WHERE clause and whether it makes the arm dead. Only foldable +// dead gates ('a'='b') are used, since the parser only prunes constant-folded WHEREs. +func genGate(rng *rand.Rand) (string, bool) { + switch rng.Intn(6) { + case 0: + return "", false // no gate (live) + case 1: + return " WHERE 1=1", false // foldable live + case 2: + return " WHERE 'a'='b'", true // foldable DEAD (calibrated → CONSTANT UInt64_0) + case 3: + return " WHERE 'q'='q'", false // foldable live + case 4: + return " WHERE x > 0", false // real predicate (live) + default: + return " WHERE x = 0", false // real predicate that must NOT be read as dead + } +} + +func genSelect(rng *rand.Rand, depth int, ancestorDead bool) (string, []occ) { + if depth <= 0 || rng.Intn(100) < 45 { + short := baseShort[rng.Intn(len(baseShort))] + clause, dead := genGate(rng) + return "SELECT x FROM " + full(short) + clause, []occ{{short, ancestorDead || dead}} + } + clause, dead := genGate(rng) + d := ancestorDead || dead + var inner string + var iocc []occ + if rng.Intn(2) == 0 { + inner, iocc = genUnion(rng, depth-1, d) + } else { + inner, iocc = genSelect(rng, depth-1, d) + } + alias := fmt.Sprintf("a%d", rng.Intn(1_000_000)) + return "SELECT x FROM (" + inner + ") " + alias + clause, iocc +} + +func genUnion(rng *rand.Rand, depth int, ancestorDead bool) (string, []occ) { + k := 2 + rng.Intn(3) + parts := make([]string, 0, k) + var occs []occ + for i := 0; i < k; i++ { + s, o := genSelect(rng, depth-1, ancestorDead) + parts = append(parts, s) + occs = append(occs, o...) + } + return strings.Join(parts, " UNION ALL "), occs +} + +func genTop(rng *rand.Rand) (string, []occ) { + depth := 2 + rng.Intn(5) // 2..6 + if rng.Intn(2) == 0 { + return genUnion(rng, depth, false) + } + return genSelect(rng, depth, false) +} + +func sets(occs []occ) (expected, referenced map[string]bool) { + expected = map[string]bool{} + referenced = map[string]bool{} + for _, o := range occs { + referenced[o.table] = true + if !o.dead { + expected[o.table] = true + } + } + return +} + +type special struct { + cat, sql string + expected []string +} + +func specials() []special { + return []special{ + {"count", "SELECT count() FROM fuzz_deps.bt0", []string{"bt0"}}, + {"join", "SELECT a.x FROM fuzz_deps.bt0 a JOIN fuzz_deps.bt1 b ON a.x=b.x", []string{"bt0", "bt1"}}, + {"self-join", "SELECT a.x FROM fuzz_deps.bt0 a JOIN fuzz_deps.bt0 b ON a.x=b.x", []string{"bt0"}}, + {"join-dead-where", "SELECT a.x FROM fuzz_deps.bt0 a JOIN fuzz_deps.bt1 b ON a.x=b.x WHERE 'a'='b'", []string{}}, + {"in-subquery", "SELECT x FROM fuzz_deps.bt0 WHERE x IN (SELECT x FROM fuzz_deps.bt1)", []string{"bt0", "bt1"}}, + {"in-subquery-dead", "SELECT x FROM fuzz_deps.bt0 WHERE 'a'='b' UNION ALL SELECT x FROM fuzz_deps.bt2 WHERE x IN (SELECT x FROM fuzz_deps.bt3)", []string{"bt2", "bt3"}}, + {"cte", "WITH c AS (SELECT x FROM fuzz_deps.bt2) SELECT x FROM c", []string{"bt2"}}, + {"cte-join", "WITH c AS (SELECT x FROM fuzz_deps.bt2) SELECT c.x FROM c JOIN fuzz_deps.bt3 t ON c.x=t.x", []string{"bt2", "bt3"}}, + {"joinGet", "SELECT joinGet('fuzz_deps.jt','v', toString(x)) AS g FROM fuzz_deps.bt0", []string{"bt0", "jt"}}, + {"joinGet-bare", "SELECT joinGet('jt','v', toString(x)) AS g FROM fuzz_deps.bt0", []string{"bt0", "jt"}}, + {"dictGet", "SELECT dictGet('fuzz_deps.mydict','val', toUInt64(x)) AS d FROM fuzz_deps.bt0", []string{"bt0", "dsrc"}}, + {"dictGetOrDefault", "SELECT dictGetOrDefault('fuzz_deps.mydict','val', toUInt64(x), '') AS d FROM fuzz_deps.bt0", []string{"bt0", "dsrc"}}, + {"dictHas", "SELECT x FROM fuzz_deps.bt0 WHERE dictHas('fuzz_deps.mydict', toUInt64(x))", []string{"bt0", "dsrc"}}, + {"cross-db", "SELECT x FROM fuzz_deps.bt0 UNION ALL SELECT x FROM fuzz_other.ext", []string{"bt0"}}, + {"cross-db-join", "SELECT a.x FROM fuzz_deps.bt0 a JOIN fuzz_other.ext b ON a.x=b.x", []string{"bt0"}}, + {"table-func", "SELECT number AS x FROM numbers(10)", []string{}}, + {"table-func-in", "SELECT x FROM fuzz_deps.bt0 WHERE x IN (SELECT number FROM numbers(5))", []string{"bt0"}}, + {"weird-name", "SELECT x FROM fuzz_deps.`weird.name`", []string{"weird.name"}}, + {"weird-name-dead", "SELECT x FROM fuzz_deps.`weird.name` WHERE 'a'='b'", []string{}}, + {"weird-name-union", "SELECT x FROM fuzz_deps.`weird.name` WHERE 1=1 UNION ALL SELECT x FROM fuzz_deps.bt0 WHERE 'a'='b'", []string{"weird.name"}}, + {"strinj-tablename", "SELECT 'table_name: fuzz_deps.bt5' AS lbl FROM fuzz_deps.bt0 WHERE x>0", []string{"bt0"}}, + {"strinj-deadconst", "SELECT x FROM fuzz_deps.bt0 WHERE s = 'constant_value: UInt64_0,'", []string{"bt0"}}, + {"strinj-where", "SELECT x FROM fuzz_deps.bt0 WHERE s = 'WHERE' OR s = 'QUERY id: 9'", []string{"bt0"}}, + {"groupby-having", "SELECT x, count() c FROM fuzz_deps.bt0 GROUP BY x HAVING c > 0 ORDER BY x LIMIT 3", []string{"bt0"}}, + {"final-prewhere", "SELECT x FROM fuzz_deps.rmt FINAL PREWHERE x > 0 WHERE s != ''", []string{"rmt"}}, + {"final", "SELECT x FROM fuzz_deps.rmt FINAL", []string{"rmt"}}, + {"final-join", "SELECT r.x FROM fuzz_deps.rmt AS r FINAL JOIN fuzz_deps.bt0 AS b ON r.x = b.x", []string{"rmt", "bt0"}}, + {"sample", "SELECT x FROM fuzz_deps.smp SAMPLE 0.5", []string{"smp"}}, + {"final-dead-arm", "SELECT x FROM fuzz_deps.rmt FINAL WHERE 'a'='b' UNION ALL SELECT x FROM fuzz_deps.bt0", []string{"bt0"}}, + {"window", "SELECT sum(x) OVER (PARTITION BY s) AS w FROM fuzz_deps.bt0", []string{"bt0"}}, + {"deep-live", "SELECT x FROM (SELECT x FROM (SELECT x FROM fuzz_deps.bt3 WHERE x>0) z WHERE 1=1) y", []string{"bt3"}}, + {"live-and-dead-same", "SELECT x FROM fuzz_deps.bt4 WHERE 1=1 UNION ALL SELECT x FROM fuzz_deps.bt4 WHERE 'a'='b'", []string{"bt4"}}, + {"all-dead-union", "SELECT x FROM fuzz_deps.bt5 WHERE 'a'='b' UNION ALL SELECT x FROM fuzz_deps.bt6 WHERE 'x'='y'", []string{}}, + {"dead-arm-with-subquery", "SELECT x FROM fuzz_deps.bt0 WHERE 1=1 UNION ALL SELECT x FROM (SELECT x FROM fuzz_deps.bt1) z WHERE 'a'='b'", []string{"bt0"}}, + {"nested-union-in-dead-arm", "SELECT x FROM fuzz_deps.bt0 WHERE 'a'='b' UNION ALL SELECT x FROM fuzz_deps.bt1 WHERE 1=1", []string{"bt1"}}, + } +} + +func toSet(ss []string) map[string]bool { + m := map[string]bool{} + for _, s := range ss { + m[s] = true + } + return m +} + +func keys(m map[string]bool) []string { + out := make([]string, 0, len(m)) + for k := range m { + out = append(out, k) + } + sort.Strings(out) + return out +} + +func TestFuzzDeps(t *testing.T) { + conn := connectFuzz(t) + defer conn.Close() + setupSchema(t, conn) + defer teardown(conn) + ctx := context.Background() + + explainDump := func(sql string) string { + rows, err := conn.Query(ctx, "EXPLAIN QUERY TREE "+sql) + if err != nil { + return " " + } + defer func() { _ = rows.Close() }() + var b strings.Builder + for rows.Next() { + var ln string + _ = rows.Scan(&ln) + b.WriteString(" " + ln + "\n") + } + return b.String() + } + + type finding struct { + cat, sql string + expected, got, under, over []string + } + var unders, overs, errs []finding + total := 0 + + check := func(cat, sql string, expected, referenced map[string]bool) { + total++ + got, err := ResolveTables(ctx, conn, fdb, sql) + if err != nil { + errs = append(errs, finding{cat: cat, sql: sql, got: []string{err.Error()}}) + return + } + gotSet := toSet(got) + var under, over []string + for tbl := range expected { + if !gotSet[tbl] { + under = append(under, tbl) + } + } + for _, g := range got { + if !expected[g] { + over = append(over, g) + } + } + sort.Strings(under) + sort.Strings(over) + if len(under) > 0 { + unders = append(unders, finding{cat, sql, keys(expected), got, under, over}) + } + if len(over) > 0 { + overs = append(overs, finding{cat, sql, keys(expected), got, under, over}) + } + } + + rng := rand.New(rand.NewSource(0xC0FFEE)) + const N = 500 + for i := 0; i < N; i++ { + sql, occs := genTop(rng) + expected, referenced := sets(occs) + check("random", sql, expected, referenced) + } + for _, sp := range specials() { + check(sp.cat, sp.sql, toSet(sp.expected), nil) + } + injPool := []string{ + "table_name: fuzz_deps.bt5", "table_name: fuzz_deps.bt0", + "function_name: joinGet", "function_name: dictGet", + "constant_value: UInt64_0,", "WHERE", "QUERY id: 0", "JOIN TREE", + } + for i, p := range injPool { + tbl := baseShort[i%len(baseShort)] + sql := fmt.Sprintf("SELECT '%s' AS lbl FROM %s WHERE x > %d", p, full(tbl), i) + check("inject", sql, toSet([]string{tbl}), nil) + } + + fmt.Printf("\n========== FUZZ DEPS REPORT ==========\n") + fmt.Printf("queries run: %d under-resolutions (BUGS): %d over-resolutions: %d explain-errors: %d\n\n", + total, len(unders), len(overs), len(errs)) + + if len(unders) > 0 { + fmt.Printf("---- UNDER-RESOLUTION (stale-cache risk) ----\n") + for _, f := range unders { + fmt.Printf("[%s] MISSED %v\n expected=%v got=%v\n sql: %s\n EXPLAIN:\n%s\n", + f.cat, f.under, f.expected, f.got, f.sql, explainDump(f.sql)) + } + } + + if len(overs) > 0 { + fmt.Printf("---- OVER-RESOLUTION (pruning miss / phantom) ----\n") + shown := 0 + for _, f := range overs { + fmt.Printf("[%s] EXTRA %v expected=%v got=%v\n sql: %s\n", f.cat, f.over, f.expected, f.got, f.sql) + if shown < 6 { + fmt.Printf(" EXPLAIN:\n%s\n", explainDump(f.sql)) + } + shown++ + } + } + + if len(errs) > 0 { + fmt.Printf("---- EXPLAIN ERRORS (would over-resolve to all tables) ----\n") + for _, f := range errs { + fmt.Printf("[%s] %v\n sql: %s\n", f.cat, f.got, f.sql) + } + } + fmt.Printf("========== END REPORT ==========\n\n") + + if len(unders) > 0 { + t.Errorf("%d under-resolution(s) found — see report above", len(unders)) + } +} diff --git a/internal/ingest/worker.go b/internal/ingest/worker.go index e663c2e1..988d485d 100644 --- a/internal/ingest/worker.go +++ b/internal/ingest/worker.go @@ -15,8 +15,8 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/mq" - "github.com/Wave-RF/WaveHouse/internal/query" "github.com/nats-io/nats.go" "github.com/nats-io/nats.go/jetstream" "go.opentelemetry.io/otel/trace" @@ -464,7 +464,7 @@ func (w *IngestWorker) handleSuccess(ctx context.Context, tableName string, msgs // every scope — and there's nothing more to add. Otherwise invalidate each // distinct scope. Doing this here (we already loop the batch once, and know it's // one table) keeps Cache.Invalidate a simple one-pass bump. - encodedTable := query.SafeEncodeNATS(tableName) + encodedTable := chsql.SafeEncodeNATS(tableName) seenScopes := make(map[string]struct{}, len(msgs)) namespaces := make([]cache.Namespace, 0, len(msgs)) @@ -479,7 +479,7 @@ func (w *IngestWorker) handleSuccess(ctx context.Context, tableName string, msgs seenScopes[pm.scope] = struct{}{} namespaces = append(namespaces, cache.Namespace{ Table: encodedTable, - Scope: query.SafeEncodeNATS(pm.scope), + Scope: chsql.SafeEncodeNATS(pm.scope), }) } diff --git a/internal/ingest/worker_test.go b/internal/ingest/worker_test.go index b0642ca1..3e8d61c9 100644 --- a/internal/ingest/worker_test.go +++ b/internal/ingest/worker_test.go @@ -17,8 +17,8 @@ import ( "time" "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/Wave-RF/WaveHouse/internal/mq" - "github.com/Wave-RF/WaveHouse/internal/query" "github.com/Wave-RF/WaveHouse/internal/testutil" "github.com/nats-io/nats.go" "github.com/nats-io/nats.go/jetstream" @@ -79,9 +79,9 @@ func makeEnvelope(t *testing.T, tableName, scope string, data map[string]any) [] // so the subject and envelope can't silently drift from the producer's contract. func newIngestMsg(t *testing.T, table, scope string, data map[string]any) *testutil.MockJetStreamMsg { t.Helper() - subj := "ingest." + query.SafeEncodeNATS(table) + subj := "ingest." + chsql.SafeEncodeNATS(table) if scope != "" { - subj += "." + query.SafeEncodeNATS(scope) + subj += "." + chsql.SafeEncodeNATS(scope) } return &testutil.MockJetStreamMsg{ MsgSubject: subj, diff --git a/internal/pipes/pipes.go b/internal/pipes/pipes.go index 69448bcf..23ef585d 100644 --- a/internal/pipes/pipes.go +++ b/internal/pipes/pipes.go @@ -13,12 +13,12 @@ import ( "strings" "sync" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/nats-io/nats.go/jetstream" ) const kvBucket = "WAVEHOUSE_PIPES" -// NamedQuery is a pre-defined SQL template with parameter support. type NamedQuery struct { Name string `json:"name"` SQL string `json:"sql"` @@ -237,9 +237,7 @@ func formatParamValue(v any) (string, error) { if isNumericLiteral(val) { return val, nil } - escaped := strings.ReplaceAll(val, `\`, `\\`) - escaped = strings.ReplaceAll(escaped, `'`, `''`) - return "'" + escaped + "'", nil + return chsql.QuoteString(val), nil case float64: // JSON numbers are float64; if it's a whole number, format as integer. if val == float64(int64(val)) { diff --git a/tests/integration/dlq_test.go b/tests/integration/dlq_test.go index 96bfa697..3c20c698 100644 --- a/tests/integration/dlq_test.go +++ b/tests/integration/dlq_test.go @@ -10,7 +10,7 @@ import ( "testing" "time" - "github.com/Wave-RF/WaveHouse/internal/query" + "github.com/Wave-RF/WaveHouse/internal/chsql" "github.com/stretchr/testify/assert" "github.com/stretchr/testify/require" ) @@ -50,7 +50,7 @@ func TestDLQ_PopulatedOnIngestWorkerFailure(t *testing.T) { // A table name that intentionally doesn't exist in ClickHouse. Per-test // suffix keeps tests independent if more DLQ tests get added later. rawTableName := fmt.Sprintf("nonexistent_table_%d", time.Now().UnixNano()) - safeTableName := query.SafeEncodeNATS(rawTableName) + safeTableName := chsql.SafeEncodeNATS(rawTableName) evt := map[string]any{ "table_name": rawTableName, @@ -94,7 +94,7 @@ func TestDLQ_PopulatedOnIngestWorkerFailureWithBadName(t *testing.T) { // Per-test suffix keeps tests independent if more DLQ tests get added later. rawTableName := fmt.Sprintf("no table.!@#&*()_=/_`%d", time.Now().UnixNano()) - safeTableName := query.SafeEncodeNATS(rawTableName) + safeTableName := chsql.SafeEncodeNATS(rawTableName) evt := map[string]any{ "table_name": rawTableName, diff --git a/tests/integration/pipe_cache_test.go b/tests/integration/pipe_cache_test.go new file mode 100644 index 00000000..2169c3f4 --- /dev/null +++ b/tests/integration/pipe_cache_test.go @@ -0,0 +1,341 @@ +//go:build integration + +package tests + +import ( + "context" + "fmt" + "net/http" + "net/http/httptest" + "testing" + "time" + + "github.com/Wave-RF/WaveHouse/internal/api" + "github.com/Wave-RF/WaveHouse/internal/auth" + "github.com/Wave-RF/WaveHouse/internal/cache" + "github.com/Wave-RF/WaveHouse/internal/chsql" + "github.com/Wave-RF/WaveHouse/internal/pipes" + "github.com/Wave-RF/WaveHouse/internal/policy" + "github.com/Wave-RF/WaveHouse/internal/testutil" + "github.com/go-chi/chi/v5" + "github.com/stretchr/testify/require" +) + +// TestPipeCacheInvalidation drives the full pipe cache path against the shared +// ClickHouse: a pipe reading A NORMAL VIEW, caching, and write-driven invalidation +// via the cascade. The pipe folds the VIEW's own namespace; the registry maps the +// view to its base table at refresh (the view carries no dependencies_table edge, +// so this must come from parsing the view's own as_select), and installs a base-> +// view cascade so a write to the base evicts the view-keyed entry. Pipes run the +// author SQL as-is (no row security), so any allowed role shares one cached result. +func TestPipeCacheInvalidation(t *testing.T) { + e := env(t) + ctx := context.Background() + + // Base table + a normal view over it. The pipe reads the VIEW; resolution + // flattens it through the tree (built from the view's parsed definition) down + // to the base table the registry knows, so a write to the base invalidates. + base := createTable(t, "user_id UInt64, org_id UInt64", "ORDER BY user_id") + require.NoError(t, e.chConn.Exec(ctx, + fmt.Sprintf("INSERT INTO `%s` SELECT number, number%%3 FROM numbers(60)", base))) + + view := base + "_v" + require.NoError(t, e.chConn.Exec(ctx, + fmt.Sprintf("CREATE VIEW `%s` AS SELECT user_id, org_id FROM `%s`", view, base))) + t.Cleanup(func() { _ = e.chConn.Exec(context.Background(), "DROP VIEW IF EXISTS `"+view+"`") }) + + // The view->base edge is discovered at refresh (createTable refreshed before + // the view existed), so refresh again now that the view is created. + require.NoError(t, e.registry.Refresh(ctx)) + + localCache, err := cache.NewLocal(1 << 20) + require.NoError(t, err) + t.Cleanup(func() { _ = localCache.Close() }) + // Install the registry's view->source cascade (normally pushed by main on each + // refresh): a pipe folds a VIEW's own namespace, so a write to the view's base + // table only evicts it via this cascade. The registry was refreshed above. + localCache.SetDependents(e.registry.Dependents()) + + store := pipes.NewMemoryStore(&pipes.NamedQuery{ + Name: "agg", SQL: fmt.Sprintf("SELECT sum(user_id) AS s FROM `%s`.`%s`", testCHDatabase, view), + AllowedRoles: []string{"viewer", "editor"}, + }) + h := api.NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), e.chConn, localCache, 30*time.Second, testutil.NopLogger()) + h.Registry = e.registry + + exec := func(role string) *httptest.ResponseRecorder { + t.Helper() + rctx := chi.NewRouteContext() + rctx.URLParams.Add("name", "agg") + c := context.WithValue(context.Background(), chi.RouteCtxKey, rctx) + c = auth.WithRole(c, role) + r := httptest.NewRequestWithContext(c, http.MethodGet, "/v1/pipes/agg/execute", nil) + w := httptest.NewRecorder() + h.Execute(w, r) + require.Equal(t, http.StatusOK, w.Code, w.Body.String()) + localCache.Wait() // ristretto Set is async + return w + } + + // First call misses (folds the view's own namespace as the dependency), then hits. + w := exec("viewer") + require.Equal(t, "MISS", w.Header().Get("X-Cache")) + require.Equal(t, `[{"s":1770}]`, w.Body.String()) // sum(0..59) + require.Equal(t, "HIT", exec("viewer").Header().Get("X-Cache")) + + // A different allowed role shares the same cached result (pipes aren't per-role). + require.Equal(t, "HIT", exec("editor").Header().Get("X-Cache")) + + // A write to the BASE table cascades to the view (registry base->view edge) and + // invalidates the view-keyed cached result. + _, err = localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(base)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec("viewer").Header().Get("X-Cache")) +} + +// TestPipeCacheInvalidation_TrivialCount proves a metadata-only count() still +// invalidates. ClickHouse answers SELECT count() FROM t from metadata (reading no +// data parts), which once defeated execution-time query_log resolution — but the +// definition-time static parse sees `FROM t` in the SQL regardless, so the table +// is a dependency and an ingest into it evicts the cached count. No EXPLAIN needed. +func TestPipeCacheInvalidation_TrivialCount(t *testing.T) { + e := env(t) + ctx := context.Background() + + base := createTable(t, "user_id UInt64", "ORDER BY user_id") + require.NoError(t, e.chConn.Exec(ctx, + fmt.Sprintf("INSERT INTO `%s` SELECT number FROM numbers(60)", base))) + + localCache, err := cache.NewLocal(1 << 20) + require.NoError(t, err) + t.Cleanup(func() { _ = localCache.Close() }) + // Install the registry's view->source cascade (normally pushed by main on each + // refresh): a pipe folds a VIEW's own namespace, so a write to the view's base + // table only evicts it via this cascade. The registry was refreshed above. + localCache.SetDependents(e.registry.Dependents()) + + store := pipes.NewMemoryStore(&pipes.NamedQuery{ + Name: "cnt", SQL: fmt.Sprintf("SELECT count() AS c FROM `%s`.`%s`", testCHDatabase, base), + AllowedRoles: []string{"viewer"}, + }) + h := api.NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), e.chConn, localCache, 30*time.Second, testutil.NopLogger()) + h.Registry = e.registry + + exec := func() *httptest.ResponseRecorder { + t.Helper() + rctx := chi.NewRouteContext() + rctx.URLParams.Add("name", "cnt") + c := context.WithValue(context.Background(), chi.RouteCtxKey, rctx) + c = auth.WithRole(c, "viewer") + r := httptest.NewRequestWithContext(c, http.MethodGet, "/v1/pipes/cnt/execute", nil) + w := httptest.NewRecorder() + h.Execute(w, r) + require.Equal(t, http.StatusOK, w.Code, w.Body.String()) + localCache.Wait() + return w + } + + w := exec() + require.Equal(t, "MISS", w.Header().Get("X-Cache")) + require.Equal(t, `[{"c":60}]`, w.Body.String()) + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // The static parse captured the counted table as a dependency, so this write + // invalidates the cached result even though the count read no data parts. + _, err = localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(base)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) +} + +// TestPipeCacheInvalidation_UnionDeadBranchPruned proves the per-binding dead-branch +// pruning: on a parameter-gated UNION, ClickHouse folds the bound predicate so the +// arm that can't match ({{source}}='web' makes the 'mobile' arm 'web'='mobile') +// resolves to a constant-false WHERE. Resolution drops that arm's table, so with +// source='web' the result depends ONLY on `web` — a write to the dead-branch `mobile` +// does NOT evict it, while a write to `web` does. This is precise, not stale: the +// pruned arm reads nothing regardless of `mobile`'s data, so it's a true non-dependency. +func TestPipeCacheInvalidation_UnionDeadBranchPruned(t *testing.T) { + e := env(t) + ctx := context.Background() + + web := createTable(t, "user_id UInt64", "ORDER BY user_id") + mobile := createTable(t, "user_id UInt64", "ORDER BY user_id") + for _, tbl := range []string{web, mobile} { + require.NoError(t, e.chConn.Exec(ctx, + fmt.Sprintf("INSERT INTO `%s` SELECT number FROM numbers(60)", tbl))) + } + + // A parameter-gated UNION; {{source}} defaults to 'web', so the 'mobile' arm's + // WHERE folds to constant-false and `mobile` is pruned from the dependency set. + sql := fmt.Sprintf( + "SELECT sum(user_id) AS s FROM ("+ + "SELECT user_id FROM `%s`.`%s` WHERE {{source:web}} = 'web' "+ + "UNION ALL "+ + "SELECT user_id FROM `%s`.`%s` WHERE {{source:web}} = 'mobile')", + testCHDatabase, web, testCHDatabase, mobile) + + localCache, err := cache.NewLocal(1 << 20) + require.NoError(t, err) + t.Cleanup(func() { _ = localCache.Close() }) + localCache.SetDependents(e.registry.Dependents()) + + store := pipes.NewMemoryStore(&pipes.NamedQuery{Name: "u", SQL: sql, AllowedRoles: []string{"viewer"}}) + h := api.NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), e.chConn, localCache, 30*time.Second, testutil.NopLogger()) + h.Registry = e.registry + + exec := func() *httptest.ResponseRecorder { + t.Helper() + rctx := chi.NewRouteContext() + rctx.URLParams.Add("name", "u") + c := context.WithValue(context.Background(), chi.RouteCtxKey, rctx) + c = auth.WithRole(c, "viewer") + r := httptest.NewRequestWithContext(c, http.MethodGet, "/v1/pipes/u/execute", nil) + w := httptest.NewRecorder() + h.Execute(w, r) + require.Equal(t, http.StatusOK, w.Code, w.Body.String()) + localCache.Wait() + return w + } + + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) // deps pruned to [web] + require.Equal(t, `[{"s":1770}]`, exec().Body.String()) // web's sum(0..59), a HIT + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // A write to the pruned dead-branch table `mobile` does NOT evict — it is a true + // non-dependency for source='web' (precise resolution, not over-resolution). + _, err = localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(mobile)}}) + require.NoError(t, err) + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // A write to the live-branch table `web` DOES evict. + _, err = localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(web)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) +} + +// TestPipeCacheInvalidation_MaterializedViewSource proves a pipe reading a +// materialized view invalidates on writes to the MV's SOURCE table. The pipe folds +// the MV's own namespace, which nothing writes directly (ingest writes the source; +// ClickHouse maintains the MV from it), so without the cascade such a pipe would be +// silently stale until its TTL. The registry maps source->MV via +// system.tables.dependencies_table and installs it as a cascade, so a source write +// fans out to evict the MV-keyed cached result. +func TestPipeCacheInvalidation_MaterializedViewSource(t *testing.T) { + e := env(t) + ctx := context.Background() + + // Source table (registry-known, ingestable). The registry refresh happens here, + // BEFORE the MV is created, so the MV is deliberately NOT a registry table — the + // realistic case the source mapping must handle. + src := createTable(t, "id UInt64, v UInt64", "ORDER BY id") + require.NoError(t, e.chConn.Exec(ctx, + fmt.Sprintf("INSERT INTO `%s` SELECT number, number FROM numbers(100)", src))) + + mv := src + "_mv" + require.NoError(t, e.chConn.Exec(ctx, fmt.Sprintf( + "CREATE MATERIALIZED VIEW `%s` ENGINE=AggregatingMergeTree ORDER BY id AS SELECT id, sumState(v) AS s FROM `%s` GROUP BY id", + mv, src))) + t.Cleanup(func() { _ = e.chConn.Exec(context.Background(), "DROP VIEW IF EXISTS `"+mv+"`") }) + + // The materialized-view -> source map is built during schema discovery, so refresh + // AFTER creating the view (createTable refreshed before it existed). + require.NoError(t, e.registry.Refresh(ctx)) + + localCache, err := cache.NewLocal(1 << 20) + require.NoError(t, err) + t.Cleanup(func() { _ = localCache.Close() }) + // Install the registry's view->source cascade (normally pushed by main on each + // refresh): a pipe folds a VIEW's own namespace, so a write to the view's base + // table only evicts it via this cascade. The registry was refreshed above. + localCache.SetDependents(e.registry.Dependents()) + + store := pipes.NewMemoryStore(&pipes.NamedQuery{ + Name: "mvpipe", SQL: fmt.Sprintf("SELECT sumMerge(s) AS s FROM `%s`.`%s`", testCHDatabase, mv), + AllowedRoles: []string{"viewer"}, + }) + h := api.NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), e.chConn, localCache, 30*time.Second, testutil.NopLogger()) + h.Registry = e.registry + + exec := func() *httptest.ResponseRecorder { + t.Helper() + rctx := chi.NewRouteContext() + rctx.URLParams.Add("name", "mvpipe") + c := context.WithValue(context.Background(), chi.RouteCtxKey, rctx) + c = auth.WithRole(c, "viewer") + r := httptest.NewRequestWithContext(c, http.MethodGet, "/v1/pipes/mvpipe/execute", nil) + w := httptest.NewRecorder() + h.Execute(w, r) + require.Equal(t, http.StatusOK, w.Code, w.Body.String()) + localCache.Wait() + return w + } + + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) // folds the MV's namespace; cascade maps source -> MV + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // A write to the SOURCE (what ingest bumps) must evict the MV-reading pipe. + _, err = localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(src)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) +} + +// TestPipeCacheInvalidation_WeirdIdentifier proves the pipe dependency-invalidation +// path operates on a table whose name a safe-identifier allowlist would reject — here +// an embedded-dot name that also LOOKS like a qualified `db.table` reference, the most +// load-bearing weird case for a feature that parses `db.table` out of EXPLAIN. It is +// the pipe-path counterpart to the query builder's identifier_roundtrip_test.go: the +// round-trip under test is that the dependency name EXPLAIN QUERY TREE reports, once +// SafeEncodeNATS-encoded, equals the namespace an ingest write to that table bumps — +// otherwise a write would silently fail to evict. createRawTable is required because +// createTable sanitizes the dots away; dotted-name resolution itself is proven in +// internal/discovery/fuzz_deps_test.go, and this carries it through the full Execute. +func TestPipeCacheInvalidation_WeirdIdentifier(t *testing.T) { + e := env(t) + ctx := context.Background() + + // Arbitrary, regex-hostile name: embedded dots so it reads as `a.b.c`. Legal in a + // quoted ClickHouse identifier; createRawTable quotes it correctly and seeds one row. + rawName := fmt.Sprintf("it_weird.dep.name_%d", tableCounter.Add(1)) + table := createRawTable(t, rawName, map[string]string{"id": "row1"}) + + localCache, err := cache.NewLocal(1 << 20) + require.NoError(t, err) + t.Cleanup(func() { _ = localCache.Close() }) + localCache.SetDependents(e.registry.Dependents()) + + store := pipes.NewMemoryStore(&pipes.NamedQuery{ + Name: "weird", + SQL: fmt.Sprintf("SELECT count() AS c FROM %s.%s", chQuoteIdent(testCHDatabase), chQuoteIdent(table)), + AllowedRoles: []string{"viewer"}, + }) + h := api.NewPipesHandler(store, policy.NewMemoryStore(&policy.Policy{}), e.chConn, localCache, 30*time.Second, testutil.NopLogger()) + h.Registry = e.registry + + exec := func() *httptest.ResponseRecorder { + t.Helper() + rctx := chi.NewRouteContext() + rctx.URLParams.Add("name", "weird") + c := context.WithValue(context.Background(), chi.RouteCtxKey, rctx) + c = auth.WithRole(c, "viewer") + r := httptest.NewRequestWithContext(c, http.MethodGet, "/v1/pipes/weird/execute", nil) + w := httptest.NewRecorder() + h.Execute(w, r) + require.Equal(t, http.StatusOK, w.Code, w.Body.String()) + localCache.Wait() + return w + } + + // First call resolves the weird-named table via EXPLAIN and folds its namespace + // into the key; second call hits. + w := exec() + require.Equal(t, "MISS", w.Header().Get("X-Cache")) + require.Equal(t, `[{"c":1}]`, w.Body.String()) + require.Equal(t, "HIT", exec().Header().Get("X-Cache")) + + // A write to the weird-named table, encoded exactly as the ingest worker encodes + // it, must evict the cached result — proving the resolved dependency namespace + // equals SafeEncodeNATS(rawName) despite the dots. + _, err = localCache.Invalidate(ctx, []cache.Namespace{{Table: chsql.SafeEncodeNATS(table)}}) + require.NoError(t, err) + require.Equal(t, "MISS", exec().Header().Get("X-Cache")) +} diff --git a/tests/integration/setup_test.go b/tests/integration/setup_test.go index 48ce527d..526c3e95 100644 --- a/tests/integration/setup_test.go +++ b/tests/integration/setup_test.go @@ -77,9 +77,15 @@ var tableCounter atomic.Uint64 func createTable(t *testing.T, columns, tableOpts string) string { t.Helper() - // Sanitize the test name into a valid CH identifier. - safe := strings.NewReplacer("/", "_", " ", "_", "-", "_").Replace(t.Name()) - name := fmt.Sprintf("it_%s_%d", strings.ToLower(safe), tableCounter.Add(1)) + safe := strings.Map(func(r rune) rune { + switch { + case r >= 'a' && r <= 'z', r >= 'A' && r <= 'Z', r >= '0' && r <= '9', r == '_': + return r + default: + return '_' + } + }, t.Name()) + name := fmt.Sprintf("it_%s_%d", safe, tableCounter.Add(1)) ctx := context.Background() stmt := fmt.Sprintf(