feat(dedupe): per-table id_field/require_id overrides

AGENTS.md

@@ -54,7 +54,7 @@ The invariant index — what must stay true. Full narrative and rationale live i
 5. **Per-table batching** — the worker groups events by table and bulk-INSERTs in schema column order; each table's batch is independent.
 6. **Dead Letter Queue** — failed batch inserts publish to `WAVEHOUSE_DLQ` (`dlq.<table>`), gated by `dlq.enabled`. No silent data loss.
 7. **Auth: always on, fail-loud, decoupled from authz (security)** — the JWT middleware always runs (no `auth.enabled`/`dev_mode` flag); it verifies with HMAC **or** JWKS (not both), with accepted `alg` pinned to the active verifier and checked before any key is used (rejects `alg:none` and cross-family confusion). No/invalid/expired token → empty role → policy `default_role`, with the bad-token reason stashed so a denying gate returns a loud `401`, not a bare `403`. Elevated access needs a valid granted role. Detail: architecture.md § `api/` + `internal/auth`; see also #11, §Security Considerations.
-8. **Optional dedup** — opt-in via `dedupe.enabled`; `dedupe.id_field` selects the JSON key.
+8. **Optional dedup** — opt-in via `dedupe.enabled`; `dedupe.id_field`/`require_id` are global defaults, overridable per table via `dedupe.tables` (an `id_field` of `""` opts a table out). Dedup keys are namespaced by table (`table` + NUL + id) so equal ids in different tables never collide — preserve when touching `internal/dedupe` (#222).
 9. **Singleflight** — `TieredCache` coalesces concurrent misses (`x/sync/singleflight`) to prevent cache stampede.
 10. **Active Sweeper** — purges NATS messages that are both ACKed (written to CH) and older than the gap window; SSE gap-fill uses `DeliverByStartTime`, no in-process ring buffer.
 11. **Hasura-style access control: fail-closed (security)** — `policy.IsAdmin` (role == `admin_role`, **exact case-sensitive**, default `"admin"`) is the single admin check, shared by `Evaluate`/`ResolveRole`/`Validate`/the `/v1/admin` gate/`RoleAllowed`. Empty/absent role matches nothing (no `"*"` wildcard); `Validate` rejects empty role keys; a `nil` policy (deleted) denies **everyone incl. admin** — a total lockout, so bootstrap from the policy file, never an implicit admin grant. `default_role` is the one sanctioned roleless exception (`ResolveRole` maps empty → it pre-eval); `default_role == admin_role` is permitted but dev-only and loudly warned (`policy.DefaultRoleGrantsAdmin`). Preserve when touching `internal/policy` (policy twin of #13; see #159). Detail: architecture.md § `policy/`.

CHANGELOG.md

@@ -11,6 +11,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Added
 
+- **Per-table dedupe `id_field` and `require_id` overrides** (`internal/config/config.go`, `internal/api/ingest.go`, `cmd/wavehouse/main.go`, `config.yaml`, `docs/src/content/docs/configuration.mdx`, plus tests in `internal/api/ingest_test.go`): closes #222. The dedupe `id_field` and `require_id` were single deployment-wide settings, forcing every table onto one id-field name and one strictness level. A new optional `dedupe.tables` map (YAML-only; the flat `dedupe.*` settings stay the defaults) overrides `id_field` and/or `require_id` per table — so one table can require a strict id while another stays lenient, or dedupe on a different field; setting a table's `id_field` to `""` skips deduplication for it. A table absent from the map inherits the defaults, so existing `dedupe.*` config is unchanged. Pairs with the table-namespaced dedupe keyspace fix (also #222, see Fixed).
+
 - **Missing-dedupe-id observability + optional strict mode** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `internal/config/config.go`, `cmd/wavehouse/main.go`, `config.yaml`, `deployments/compose/standalone.yaml`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #219. With dedupe enabled, a row missing the configured `id_field` can't be deduped — previously it was published with idempotency silently disabled and *no* log or metric, so a producer bug that dropped the id turned off the guarantee for those rows unnoticed. Now every such row is logged at `WARN` and counted by a new `wavehouse_ingest_dedupe_missing_id_total` counter (labeled by `table`), making the loss observable server-side. A new opt-in `dedupe.require_id` (`WH_DEDUPE_REQUIRE_ID`, default `false`) turns that signal into enforcement: a row missing the id is rejected (`400` for a single insert; a per-record failure in a batch) instead of published — a tripwire for producers that must guarantee the id (complements the client-side [#202](https://github.com/Wave-RF/WaveHouse/issues/202)). Default behavior is unchanged.
 - **"Durability & Storage" operations guide** (`docs/src/content/docs/durability.md` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/reverse-proxy.mdx`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/deployment.md`): documents #84. A new Operations page making the embedded-JetStream durability contract explicit before the docs site publishes: a `200` from `POST /v1/ingest` means the event has been `fsync`'d to disk on the node (the server runs with `SyncAlways: true` in `internal/mq/embedded.go`), which makes the storage substrate's `fsync` tail the ingest latency floor. Covers the contract (and how it differs from JetStream's default page-cache-then-periodic-sync mode), why a slow `fsync` tail manifests as `create stream: ... context deadline exceeded` and `503` backpressure, a where-it's-cheap-vs-expensive substrate table (managed cloud block storage and PLP NVMe vs. ZFS-without-SLOG / qcow2-on-`ext4` / spinning disks), an `fio` recipe + verdict bands to measure your own storage (with the macOS `F_FULLFSYNC` honesty caveat), and the symptom checklist. Forward-references the configurable group-commit interval (`mq.sync_interval`, [#139](https://github.com/Wave-RF/WaveHouse/issues/139)) and the planned `wavehouse storage-check` preflight ([#84](https://github.com/Wave-RF/WaveHouse/issues/84)) without claiming either exists yet. Cross-linked from Configuration (Message Queue), Deployment (Persistent Storage), and the Ingest Pipeline's worker-side ack section; no code changes.
 - **"Behind a reverse proxy" deployment guide** (`docs/src/content/docs/reverse-proxy.mdx` (new), `docs/src/config/sidebar.ts`, `docs/src/content/docs/deployment.md`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `internal/api/stream.go`, `internal/api/stream_test.go`): closes #241. A new Operations page for the common "WaveHouse behind nginx / Caddy / Cloudflare Tunnel" setup, since several behaviors only matter behind a proxy and weren't documented together. Covers: TLS termination (WaveHouse serves plain HTTP and manages no certs); the request-body size limits and the division of responsibility (WaveHouse ships fixed in-code memory-safety backstops — 1 MiB control / 16 MiB ingest — while the proxy is the tunable *outer* limit, so a missing/loose proxy limit can't OOM the server); Server-Sent Events buffering + idle-timeout tuning (WaveHouse sends a `: connected` comment on open plus a periodic `:` keepalive comment so quiet streams survive proxy idle timeouts, [#226](https://github.com/Wave-RF/WaveHouse/issues/226)); the `?token=` / `since` / `Last-Event-ID` forwarding streams need; `X-Forwarded-For` trust (don't expose `:8080` directly — it's honored, so a direct client could spoof it); and which health paths to expose (`/livez`/`/readyz` internal-optional, `/v1/health` must stay public). Ships full example nginx, Caddy, and Cloudflare-Tunnel configs, and is cross-linked from Deployment, Configuration, and the API reference. One small code change lands with it: the SSE endpoint (`GET /v1/stream`) now sets `X-Accel-Buffering: no` so nginx-class proxies stream events without buffering out of the box (nginx strips the header before the client sees it; Caddy/Cloudflare ignore it). The health-probe guidance is also upgraded from "optional" to a recommendation — keep the bare `/livez`/`/readyz`/`/healthz` paths internal (a public `/readyz` turns each hit into a ClickHouse `Ping`) and expose only `/v1/health` publicly.
@@ -28,6 +30,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 - **Dependabot npm updates moved to the pnpm workspace root, fixing recurring `ERR_PNPM_OUTDATED_LOCKFILE` CI failures on dependency PRs** (`.github/dependabot.yml`, `docs/src/content/docs/development.md`, `SECURITY.md`): the three per-member npm update configs (`directory: /docs`, `/clients/ts`, `/tests/e2e/sdk`) are replaced by a single config at the workspace root (`directory: /`, group `npm-deps`, prefix `deps`). The repo has one root `pnpm-lock.yaml` (the #190 consolidation), and Dependabot only updates a lockfile co-located with the manifest it targets — so a per-member config bumped a member's `package.json` without regenerating the root lockfile, and every such PR (e.g. #211, #337) then failed CI's `pnpm install --frozen-lockfile` with `ERR_PNPM_OUTDATED_LOCKFILE`; no rebase could fix it, because a rebase never regenerates the lockfile. Pointing at the root lets Dependabot read `pnpm-workspace.yaml`, walk every member, and update the one lockfile within the PR, and as a bonus brings the root `package.json`'s own devDeps (biome, markdownlint, nyc) under Dependabot. Trade-off: one combined weekly npm PR with a single `deps:` prefix instead of three per-area PRs (`docs:` / `deps(sdk):` / `deps(tests):`).
 
+### Fixed
+
+- **Dedupe no longer collapses explicit-null `id_field` values onto one key, and `require_id` now rejects them** (`internal/api/ingest.go`, `internal/api/ingest_test.go`, `docs/src/content/docs/configuration.mdx`, `docs/src/content/docs/api.md`, `docs/src/content/docs/architecture.md`): closes #370. The dedupe check keyed on field *presence*, so an explicit JSON `null` id (`{"event_id": null}`) — a present key with no value — was treated as a real id and deduped on `fmt.Sprint(nil)`, the literal string `"<nil>"`. Every null-id row across every table collided onto that one key (all but the first silently dropped as duplicates), and the row was invisible to both `require_id` strict mode and the `wavehouse_ingest_dedupe_missing_id_total` counter. A `null` `id_field` is now treated exactly like a missing one — logged, counted, and rejected under `require_id` (or published un-deduped otherwise). Scoped to `null`; an empty string stays a real value that dedupes. Reachable whenever the `id_field` column is `Nullable(...)` or has a `DEFAULT`, which `discovery.Validate` lets a null pass through.
+- **Dedupe keys are now namespaced by table, closing a silent cross-table collision** (`internal/dedupe/dedupe.go`, `internal/dedupe/embedded.go`, `internal/api/ingest.go`, plus tests in `internal/dedupe/embedded_test.go`, `internal/testutil/mocks.go`): part of #222. `CheckAndMark` keyed Pebble on the raw id value with no table prefix, so the same id *value* ingested into two different tables collided — the second row was silently dropped as a "duplicate." The key is now `table\x00id` (a ClickHouse identifier can't contain a NUL byte, so the table prefix can't be forged by an id value). **Breaking:** the on-disk key format changes, so existing dedupe state no longer matches after upgrade — every event is treated as new for one round. Best-effort idempotency means no stored data is lost, but a duplicate submitted across the upgrade boundary won't be caught.
+
 ### Security
 
 - **Policy `_in` is now enforced on both the row-`filter` and insert-`check` paths, closing a fail-open row-security gap** (`internal/policy/policy.go`, `internal/api/ingest.go`, `docs/src/content/docs/access-control.mdx`, plus tests in `internal/policy/policy_test.go`, `internal/api/ingest_test.go`): closes #224. The `Filter` schema accepted `_in` but the engine never read it: on the row-`filter`/SELECT path `resolveFilters` had no `_in` branch, so a row-security filter like `tenant_id: { _in: … }` produced **no `WHERE` predicate** and the role saw every row instead of its tenant subset (a fail-open, same family as #223); on the `check`/INSERT path only `_eq` was honored, silently dropping any other operator. `_in` now takes a single claim that resolves to a JSON **array** (the multi-tenant case — a token's `tenant_ids` list) and emits `col IN (?, …)` with one bound param per element; a scalar claim is a one-element set, and an empty/absent claim matches **no rows** (fail-closed) rather than widening to all of them. On the insert path an `_in` check requires the column be present and one of the set — there is no single value to auto-inject as `_eq` does, so an omitted column is rejected (`403 check failed`). The comparison operators are enforced on `filter` (`_eq`/`_neq`/`_gt`/`_lt`/`_in` all produce predicates now, so nothing is rejected there) and, on `check`, `_neq`/`_gt`/`_lt` become a loud config-load rejection (no insert-time semantics; `check` honors `_eq` + `_in`). The `_in` value stays a single templated string in the wire schema (Go `Filter.In`, SDK `PolicyFilter._in`), matching the established "set = array" shape of the caller-query `in` operator.

cmd/wavehouse/main.go

@@ -331,6 +331,13 @@ func run() int {
 		ingestHandler.Dedup = dedup
 		ingestHandler.IDField = cfg.Dedupe.IDField
 		ingestHandler.RequireID = cfg.Dedupe.RequireID
+		if len(cfg.Dedupe.Tables) > 0 {
+			overrides := make(map[string]api.DedupeOverride, len(cfg.Dedupe.Tables))
+			for t, o := range cfg.Dedupe.Tables {
+				overrides[t] = api.DedupeOverride{IDField: o.IDField, RequireID: o.RequireID}
+			}
+			ingestHandler.DedupeTables = overrides
+		}
 	}
 
 	var dlqHandler *api.DLQHandler

config.yaml

@@ -73,6 +73,16 @@ dedupe:
   # Reject rows missing id_field instead of publishing them un-deduped. Off by
   # default (they are logged + counted via wavehouse_ingest_dedupe_missing_id_total).
   require_id: false
+  # Per-table overrides of id_field / require_id (YAML-only; the settings above
+  # are the defaults). A table not listed inherits the defaults; set a table's
+  # id_field to "" to skip dedupe for it. (#222)
+  # tables:
+  #   payments:
+  #     require_id: true      # payments must carry an id or be rejected
+  #   page_views:
+  #     id_field: view_id     # dedupe this table on view_id instead of event_id
+  #   audit_log:
+  #     id_field: ""          # skip dedupe for this table entirely
 
 cache:
   l1_max_cost: 67108864

docs/src/content/docs/api.md

@@ -225,7 +225,7 @@ The body is a **flat JSON object** whose keys must match column names in the tar
 | ------ | ---- | ----- |
 | 400 | `{"error":"invalid json"}` | Malformed request body |
 | 400 | `{"error":"unknown column ... for table ..."}` (also: `missing required column ...`, `type mismatch for column ...`, `null value for non-nullable column ...`) | Schema validation failure (unknown fields, type mismatches, missing required columns, null in a non-nullable column). The body is the validator's message verbatim — there is no `validation failed:` prefix. |
-| 400 | `{"error":"missing dedupe id field \"event_id\""}` | Only when dedupe is enabled with `dedupe.require_id: true` and the row lacks the configured `id_field`. With `require_id: false` (the default) the row is instead published un-deduped. Either way — reject or publish — the row is logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total`. In a batch this is a per-record failure, not a whole-request error. |
+| 400 | `{"error":"missing dedupe id field \"<id_field>\""}` (`<id_field>` is the table's effective dedupe field — the `dedupe.id_field` default, e.g. `event_id`, or a `dedupe.tables` per-table override) | Only when dedupe is enabled and `require_id` is in effect for the table (the global `dedupe.require_id`, or a per-table `dedupe.tables` override) and the row lacks that `id_field` — whether the field is missing or sent as an explicit JSON `null`. With `require_id` off (the default) the row is instead published un-deduped. Either way — reject or publish — the row is logged at `WARN` and counted by `wavehouse_ingest_dedupe_missing_id_total`. In a batch this is a per-record failure, not a whole-request error. |
 | 401 | `{"error":"invalid token"}` / `{"error":"token expired"}` | A present-but-invalid/expired token was supplied and denied (the gate surfaces the token reason rather than silently falling back to `default_role`) |
 | 403 | `{"error":"forbidden"}` (empty-role variant: `forbidden: request has no role and no public default_role is configured`) | The resolved role lacks `insert` on the table |
 | 404 | `{"error":"unknown table: ..."}` | Table not found in ClickHouse schema |