Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actions/setup-env/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ runs:
# ask pnpm for the actual store path.
- uses: pnpm/action-setup@0e279bb959325dab635dd2c09392533439d90093 # v6.0.8
with:
version: "10.33.0"
version: "11.1.3"
run_install: false

- name: Resolve pnpm store directory
Expand Down
1 change: 1 addition & 0 deletions .nvmrc
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
22
7 changes: 4 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,9 +130,10 @@ Tooling notes:
auto-installed to `.bin/<os>_<arch>/` on first `make lint` (or via
`make tools`). Not in `go.mod` — its dependency tree conflicts with the
main module.
- `pnpm` (>= 10.33) and `Node.js` (>= 20) must be on your PATH; the SDK and
E2E test harnesses both shell out to `pnpm`. `make tools` runs
`pnpm install --frozen-lockfile` in `clients/ts/` and `tests/e2e/sdk/`.
- `pnpm` (>= 11.1) and `Node.js` (22 LTS — pinned via `.nvmrc` at the repo
root, matches CI) must be on your PATH; the SDK and E2E test harnesses
both shell out to `pnpm`. `make tools` runs `pnpm install
--frozen-lockfile` in `clients/ts/` and `tests/e2e/sdk/`.
Comment thread
EricAndrechek marked this conversation as resolved.
- `GNU Make 4+` is required (uses `--output-sync=target`); macOS ships BSD
Make 3.81 which will not parse the Makefile. See `docs/development.md` §
Prerequisites for the full setup checklist.
Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,11 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- WebSocket internal dispatch carries the subscribing table name through the merged channel instead of re-unmarshalling each payload to extract `table_name`. Drops the per-message redundant unmarshal and — for non-`EventMessage` payloads (raw-JSON pass-through) — keeps the outbound envelope's `"table"` field labelled with the actual subscription instead of falling back to `""`. No behaviour change for the standard `EventMessage` path.
- Doc-sync for the `?topic=` → `?table=` rename: `docs/src/content/docs/getting-started.md` quick-start now shows `?table=clicks` (the previous `?topic=ingest.clicks` example would 400 against the new contract), and `docs/src/content/docs/sdk.md` `SharedWSManager` section now describes per-table (not per-topic) subscriptions and notes that the legacy NATS-style wildcards are no longer accepted server-side.
- **Boot-time schema discovery no longer crashes the binary** (`cmd/wavehouse/main.go`, `internal/api/health.go`, `internal/discovery/discovery.go`, `docs/src/content/docs/api.md`, `docs/src/content/docs/deployment.md`). Previously, any error from the initial `SchemaRegistry.Refresh` — connection-refused, missing database, transient network blip — caused `cmd/wavehouse` to call `os.Exit(1)`. The supervisor would restart the process every ~10s in an unbounded loop, port 8080 never bound, and operators got `connection refused` on probes even when ClickHouse was otherwise healthy. Now the first Refresh failure is non-fatal: a new `api.BootState` is set with the diagnostic, the server still binds `:8080`, and a background goroutine calls the new `SchemaRegistry.RetryRefresh(ctx, 2s, 60s, onAttempt)` with exponential backoff until success or shutdown. While `BootState.Err()` is non-nil, `/health` returns 503 with `{"status":"degraded","error":"…"}` so an operator can `curl /health` to learn why the gateway isn't ready to serve traffic instead of grepping a restart-loop log; `/ready` returns 503 with `{"status":"not ready",…}` for the same reason. Once `BootState.Set(nil)` fires (either from the initial sync `Refresh` or from `SchemaRegistry.RetryRefresh` success), `/health` stays at `200 OK` for the rest of the process lifetime — that flip is sticky and reflects "boot completed once." `/ready`, by contrast, remains conditional on current ClickHouse reachability: it can flip back to `503` on transient runtime blips after boot, since readiness is a probe of "can I serve traffic right now," not "did I ever boot." Resolves #95.
- **Frontend toolchain pinned: pnpm 10.33 → 11.1.3, Node 22 LTS via `.nvmrc`** (`.nvmrc`, `.github/actions/setup-env/action.yml`, `clients/ts/package.json`, `clients/ts/pnpm-workspace.yaml`, `docs/package.json`, `docs/pnpm-workspace.yaml`, `tests/e2e/sdk/package.json`, `tests/e2e/sdk/pnpm-workspace.yaml`, `AGENTS.md`, `README.md`, `docs/src/content/docs/development.md`). Closes #160. Three TS workspaces (`clients/ts`, `tests/e2e/sdk`, `docs`) plus the `setup-env` composite action bump from `pnpm@10.33.0` to `pnpm@11.1.3`. pnpm 11 deprecated the `pnpm` field in `package.json` — `onlyBuiltDependencies` moved into a new per-workspace `pnpm-workspace.yaml` and was renamed to `allowBuilds:` with a map-of-booleans syntax (each workspace ships its own file even though they're not a single monorepo — having them in-tree shadows any stray ancestor `pnpm-workspace.yaml`, which is how pnpm 11 walks the directory tree). `clients/ts` and `tests/e2e/sdk` allow `esbuild`; `docs` additionally allows `sharp` (Astro image pipeline) and `workerd` (Cloudflare Workers runtime). New repo-root `.nvmrc` pins Node 22 LTS — matches the `setup-env` action's existing `node-version: "22"` — and `docs/package.json#engines.node` bumps from `>=20.0` to `>=22.0` so corepack / volta warn local devs before CI does. The original issue's open question about `electron-winstaller` resolved to a contamination from a stray `pnpm-workspace.yaml` in the developer's home directory, not a real WaveHouse transitive — correctly omitted from `allowBuilds`. The Node 26 Vitest crash mentioned in the issue is no longer reproducible after the recent `vitest@^4` bump in `clients/ts` (#103); the `.nvmrc` still lands because pinning has independent value (CI parity + Node-major-surprise guard).

### Security

- **`minimumReleaseAge: 10080` (7 days) on every pnpm workspace** (`clients/ts/pnpm-workspace.yaml`, `docs/pnpm-workspace.yaml`, `tests/e2e/sdk/pnpm-workspace.yaml`): pnpm 11 will refuse to install any package published in the last seven days, giving npm and security researchers time to flag a compromised release before it lands in our lockfile. Existing locked versions are grandfathered. For a one-off override on an urgent hotfix release, list the package under `minimumReleaseAgeExclude:` in the same file. Part of #160.

### Fixed
- **Ingest worker no longer infinite-retries permanent delete errors** (`internal/ingest/bento.go`, `internal/ingest/bento_test.go`, `docs/src/content/docs/architecture.md`, `AGENTS.md`): `jsInput.Read`'s `action: "delete"` block called `m.Nak()` on every `chConn.Exec` failure, which JetStream interprets as "redeliver immediately." A delete whose error was *deterministic* (syntax error, unknown table, malformed identifier) would loop forever — clogging the buffer consumer, burning CPU, and spamming logs with the same message. Phase 1 of issue #91: every delete-Exec error is now treated as permanent. The original NATS envelope is published to `dlq.<table>` (reusing the existing `bentoDLQDropped` counter when the DLQ publish itself fails) and the message is `DoubleAck`'d so it leaves the main queue. Issue #91 stays open after this lands as the Phase 2 tracker for transient-vs-permanent error classification (timeouts and network errors should still `Nak()` for retry); Phase 1 alone is the stopgap, Phase 2 makes the trade-off acceptable in production.
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,7 @@ WaveHouse is built as an application, not a library — `internal/` packages are

## 💻 Local Development

You'll need **Go 1.26+, GNU Make 4+, Docker (or Podman) with Compose v2, Node.js 20+, and pnpm 10+** on your PATH — see [docs/development.md § Prerequisites](docs/development.md#prerequisites) for the full list, version requirements, and macOS gotchas (BSD Make 3.81 won't work).
You'll need **Go 1.26+, GNU Make 4+, Docker (or Podman) with Compose v2, Node.js 22 LTS (pinned via `.nvmrc`, matches CI), and pnpm 11.1+** on your PATH — see [docs/development.md § Prerequisites](docs/development.md#prerequisites) for the full list, version requirements, and macOS gotchas (BSD Make 3.81 won't work).

For building and testing WaveHouse locally with hot-reload:

Expand Down
9 changes: 2 additions & 7 deletions clients/ts/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"name": "@wavehouse/sdk",
"version": "0.1.0",
"description": "TypeScript SDK for WaveHouse — schema-aware real-time API gateway for ClickHouse",
"packageManager": "pnpm@10.33.0",
"packageManager": "pnpm@11.1.3",
"type": "module",
"main": "./dist/index.cjs",
"module": "./dist/index.js",
Expand Down Expand Up @@ -48,10 +48,5 @@
"sse",
"query-builder",
"typescript"
],
"pnpm": {
"onlyBuiltDependencies": [
"esbuild"
]
}
]
}
21 changes: 21 additions & 0 deletions clients/ts/pnpm-workspace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
# pnpm 11+ moved settings out of package.json#pnpm into this file.
# This is a single-package "workspace" — only used for settings, no
# `packages:` field. (Each WaveHouse pnpm project ships its own copy
# of this file; they're not one shared monorepo.)

# Build scripts pnpm is allowed to run during install. Without this,
# pnpm 11 errors out with ERR_PNPM_IGNORED_BUILDS on any postinstall
# script for an unlisted package.
# esbuild — postinstall fetches the platform-native binary; tsup +
# vitest both depend on it.
# To add a new entry: `pnpm approve-builds <pkg>` (interactive) or
# edit this file directly with `<pkg>: true`.
allowBuilds:
esbuild: true

# Refuse packages published in the last 7 days (10080 minutes) so npm
# has time to surface compromised releases before they land in our
# lockfile. Use `minimumReleaseAgeExclude: [pkg-name]` for one-off
# overrides if a hotfix release needs to land before the window
# expires.
minimumReleaseAge: 10080
11 changes: 2 additions & 9 deletions docs/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
"type": "module",
"version": "0.0.1",
"private": true,
"packageManager": "pnpm@10.33.0",
"packageManager": "pnpm@11.1.3",
"scripts": {
"dev": "astro dev",
"start": "astro dev",
Expand Down Expand Up @@ -32,13 +32,6 @@
"wrangler": "^4.81.0"
},
"engines": {
"node": ">=20.0"
},
"pnpm": {
"onlyBuiltDependencies": [
"esbuild",
"sharp",
"workerd"
]
"node": ">=22.0"
}
}
12 changes: 12 additions & 0 deletions docs/pnpm-workspace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
# See clients/ts/pnpm-workspace.yaml for full rationale. Kept in sync
# across all three WaveHouse pnpm projects manually.
Comment thread
EricAndrechek marked this conversation as resolved.
#
# Docs needs extra build scripts beyond esbuild:
# sharp — native image pipeline used by Astro's image optimizer.
# workerd — Cloudflare Workers runtime fetched by wrangler.
allowBuilds:
esbuild: true
sharp: true
workerd: true

minimumReleaseAge: 10080
8 changes: 4 additions & 4 deletions docs/src/content/docs/development.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@ You need these on your `PATH` before any `make` recipe will work end-to-end:
| **GNU Make** | **4.0+** | The Makefile uses `--output-sync=target` (Make 4 only) and bash-pinned recipes. macOS ships with BSD Make 3.81, which **will not work** | macOS: `brew install make` then use `gmake` or put `$(brew --prefix make)/libexec/gnubin` on your PATH. Linux: usually already installed |
| **bash** | 4+ recommended | Recipes are pinned to `bash`; the helper scripts under `scripts/` use `set -euo pipefail` and bash arrays | macOS default is bash 3.2 (works for current recipes, but `brew install bash` is safer); Linux distros ship 4+ |
| **Docker** *(or Podman)* | Engine 20.10+ with the Compose **v2** plugin (`docker compose`, no hyphen) | Compose stacks under `deployments/compose/` and `tests/e2e/compose.yaml`; integration tests boot a ClickHouse testcontainer | [Docker Desktop](https://docs.docker.com/get-docker/), [colima](https://github.com/abiosoft/colima), or [Podman](https://podman.io) with `podman-compose` / the `podman compose` plugin. The testcontainers Go library also honors `DOCKER_HOST` for rootless Podman setups |
| **Node.js** | 20+ | Runtime for pnpm and the Vitest suites | [nodejs.org](https://nodejs.org/) or `nvm`/`fnm`/`volta` |
| **pnpm** | 10.33+ (pinned via `packageManager` in `clients/ts/package.json`, `tests/e2e/sdk/package.json`, and `docs/package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site; `make build-sdk`, `make test-sdk`, `make test-e2e`, `make build-docs`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` | `corepack enable && corepack prepare pnpm@10.33.0 --activate` (recommended), or `npm i -g pnpm` |
| **Node.js** | 22 LTS — pinned via `.nvmrc` at the repo root | Runtime for pnpm and the Vitest suites. Pinned to match CI (`setup-node` uses 22) and to avoid Node-major surprises; older Vitest versions in this repo were known to crash on Node 26 with a V8 heap-allocation abort | [nodejs.org](https://nodejs.org/) or `nvm use` / `fnm use` / `volta` (all read `.nvmrc`) |
| **pnpm** | 11.1+ (pinned via `packageManager` in `clients/ts/package.json`, `tests/e2e/sdk/package.json`, and `docs/package.json`) | Package manager for the TypeScript SDK, E2E test harness, and docs site; `make build-sdk`, `make test-sdk`, `make test-e2e`, `make build-docs`, `make dev-docs`, `make preview-docs` all shell out to `pnpm` | `corepack enable && corepack prepare pnpm@11.1.3 --activate` (recommended), or `npm i -g pnpm` |
| **git** + **curl** | any recent | `git` for source + version metadata in builds; `curl` is used by the Makefile to fetch the pinned `golangci-lint` binary into `.bin/` | usually preinstalled |

### Auto-installed by `make tools`
Expand All @@ -36,8 +36,8 @@ Run `make tools` once after cloning to populate everything that doesn't have to
go version # go1.26+
make --version # GNU Make 4.x
docker compose version
node --version # v20+
pnpm --version # 10.33+
node --version # v22.x (matches .nvmrc and CI)
pnpm --version # 11.1+
```

If any of those are wrong/missing, the Makefile recipes will fail with confusing errors (e.g. `--output-sync` is unrecognized on Make 3.81; `pnpm: command not found` on `make test-sdk`).
Expand Down
7 changes: 1 addition & 6 deletions tests/e2e/sdk/package.json
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"private": true,
"name": "wavehouse-e2e",
"packageManager": "pnpm@10.33.0",
"packageManager": "pnpm@11.1.3",
"type": "module",
"scripts": {
"test": "vitest run",
Expand All @@ -14,10 +14,5 @@
"eventsource": "^4.1.0",
"typescript": "^5.5.0",
"vitest": "^3.0.0"
},
"pnpm": {
"onlyBuiltDependencies": [
"esbuild"
]
}
}
7 changes: 7 additions & 0 deletions tests/e2e/sdk/pnpm-workspace.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# See clients/ts/pnpm-workspace.yaml for full rationale. Kept in sync
# across all three WaveHouse pnpm projects manually.

allowBuilds:
esbuild: true

minimumReleaseAge: 10080
Loading