Area: infra · ops · reliability · storage — footgun/gap · found via WaveHouse-Stats dogfooding (lived incident)
Expected: one table's heavy/failing ClickHouse merges shouldn't be able to silently fill the data volume and make the whole gateway unbootable; and when the node does go down for lack of disk, the surfaced signal should name the cause (not a bare 502).
Actual: a ClickHouse merge-failure loop (trigger this round: a native-JSON column whose ~1,300 distinct paths made merges exceed max_server_memory_usage on a 4 GiB box — but the chain applies to any sustained merge failure / disk pressure) retries ~every 2 s and self-amplifies by bloating ClickHouse's own diagnostic tables — system.trace_log + system.text_log grew to ~24 GiB and filled a 30 GiB volume to 96%. WaveHouse's embedded NATS JetStream then can't reserve storage on restart — internal/mq/embedded.go:104-110 (MaxBytes reservation → create stream: … insufficient storage resources available) — so the process never binds :8080 and Caddy returns a bare 502 with nothing pointing at "disk full / merges failing." WaveHouse sets no TTL on ClickHouse's *_log tables (no such handling anywhere in internal/).
Impact: total, undiagnosable outage. In the WaveHouse-Stats incident: ~145k merge failures / 3 days; root-causing required SSH + du + system.part_log (no readiness/health signal named the cause), and the migration to repair it could not run through WaveHouse's own admin API — reading the heavy column OOMs the memory-tight box, so the 502 gateway blocks its own repair path (ClickHouse itself was fine via direct clickhouse-client). Sharpest for a managed WaveHouse Cloud: a tenant's column-type choice on a constrained instance → total outage with no backpressure/alert, not even repairable through the product API.
Note: consumer suggestions (named "data volume full" readiness signal, a merge-retry circuit-breaker, a TTL on CH's *_log tables, keeping admin/query usable under memory pressure) live in the source doc — fix belongs in the PR, not here.
Related: #138 (NATS MaxBytes disk over-reservation), #236 (native-JSON ingest gap), #326 (DoS / resource-exhaustion hardening epic), #94 (saturation observability). Root of the merge OOM — CH max_memory_usage server caps — was #316 (fixed in #335), but that guards query-path allocation, not background-merge failure or the disk-fill/boot chain.
From WaveHouse-Stats WAVEHOUSE-FEEDBACK.md dogfooding (lived native-JSON disk-fill incident, 2026-06-30); validated by code-read against 029fe1f on 2026-07-01 (internal/mq/embedded.go storage reservation confirmed; no CH *_log TTL handling in internal/).
Area: infra · ops · reliability · storage — footgun/gap · found via WaveHouse-Stats dogfooding (lived incident)
Expected: one table's heavy/failing ClickHouse merges shouldn't be able to silently fill the data volume and make the whole gateway unbootable; and when the node does go down for lack of disk, the surfaced signal should name the cause (not a bare 502).
Actual: a ClickHouse merge-failure loop (trigger this round: a native-
JSONcolumn whose ~1,300 distinct paths made merges exceedmax_server_memory_usageon a 4 GiB box — but the chain applies to any sustained merge failure / disk pressure) retries ~every 2 s and self-amplifies by bloating ClickHouse's own diagnostic tables —system.trace_log+system.text_loggrew to ~24 GiB and filled a 30 GiB volume to 96%. WaveHouse's embedded NATS JetStream then can't reserve storage on restart —internal/mq/embedded.go:104-110(MaxBytesreservation →create stream: … insufficient storage resources available) — so the process never binds:8080and Caddy returns a bare 502 with nothing pointing at "disk full / merges failing." WaveHouse sets no TTL on ClickHouse's*_logtables (no such handling anywhere ininternal/).Impact: total, undiagnosable outage. In the WaveHouse-Stats incident: ~145k merge failures / 3 days; root-causing required SSH +
du+system.part_log(no readiness/health signal named the cause), and the migration to repair it could not run through WaveHouse's own admin API — reading the heavy column OOMs the memory-tight box, so the 502 gateway blocks its own repair path (ClickHouse itself was fine via directclickhouse-client). Sharpest for a managed WaveHouse Cloud: a tenant's column-type choice on a constrained instance → total outage with no backpressure/alert, not even repairable through the product API.Note: consumer suggestions (named "data volume full" readiness signal, a merge-retry circuit-breaker, a TTL on CH's
*_logtables, keeping admin/query usable under memory pressure) live in the source doc — fix belongs in the PR, not here.Related: #138 (NATS
MaxBytesdisk over-reservation), #236 (native-JSONingest gap), #326 (DoS / resource-exhaustion hardening epic), #94 (saturation observability). Root of the merge OOM — CHmax_memory_usageserver caps — was #316 (fixed in #335), but that guards query-path allocation, not background-merge failure or the disk-fill/boot chain.From WaveHouse-Stats
WAVEHOUSE-FEEDBACK.mddogfooding (lived native-JSON disk-fill incident, 2026-06-30); validated by code-read against029fe1fon 2026-07-01 (internal/mq/embedded.gostorage reservation confirmed; no CH*_logTTL handling ininternal/).