ADR-0003: Fix swallowed startup error logging

[nicholas-ruest]

Implements ADR-0003. Source changes only — this PR deploys nothing and sets no environment variable on any service.

What was broken

Cloud Run revision ruvvector-service-00052-dxg crash-looped on startup and the entire diagnostic payload was {"error":{}}.

(a) Every error log in the repo was empty. Pino applies serializers by property name. src/utils/logger.ts registered pino.stdSerializers.err under the key err, but this codebase logs caught values under error — at 62 call sites, including every startup-hardening path (healthCheck.ts, DatabaseClient.ts, immutability.ts, and the top-level catch at index.ts:635). Error.prototype.message/.stack/.name are non-enumerable, so JSON.stringify(new Error('boom')) === '{}'. The service was undebuggable at startup — precisely where it is designed to crash hard.

(b) The actual crash was an unset RUVVECTOR_EMBEDDING_DIM, and the assertion whose job is to report missing required variables was structurally incapable of catching it: REQUIRED_ENV_VARS_PRODUCTION had five entries and the dimension was not among them. It logged "missing":0 while the service was already doomed two assertions later.

What changed

File	Change
src/utils/logger.ts	Register error and reason alongside err in the serializer map
src/startup/assertions.ts	Add RUVVECTOR_EMBEDDING_DIM to REQUIRED_ENV_VARS_PRODUCTION
tests/unit/logger-serialization.test.ts	New regression test (7 cases)
docs/adr/ADR-0003-*.md	Status → Implemented; verification output recorded

One serializer entry fixes all 62 call sites atomically, with no chance of missing one. stdSerializers.err emits type/message/stack, walks cause chains, and passes non-Error values through unchanged. The existing err key is retained and still works.

reason gets the same treatment rather than editing index.ts:614, so unhandled rejections are legible too. It is used as a log key in exactly one place, so the map entry is not overbroad.

Rejected: pino's errorKey: 'error' option, which configures exactly this behavior but requires pino 9.0. This service is on 8.21.0; an incident fix should not carry a major version bump.

Real test output

npm test — 130 passed, 8 failed, 138 total.

The 8 failures are pre-existing on main and unrelated. Verified by stashing this branch's changes and running the same two suites at 1813a25: identical 8 failed, 34 passed in executions.test.ts and simulations.test.ts. This branch is exactly +7 passing tests.

New regression suite:

PASS tests/unit/logger-serialization.test.ts
  logger error serialization
    ✓ serializes an Error logged under the "error" key (23 ms)
    ✓ reports the concrete error class rather than a bare "Error" (4 ms)
    ✓ serializes an Error logged under the "reason" key (2 ms)
    ✓ walks the cause chain (3 ms)
    ✓ passes non-Error values through untouched (2 ms)
    ✓ still serializes the "err" key (2 ms)
    ✓ demonstrates the swallowed-error bug when "error" is not registered (3 ms)

Tests: 7 passed, 7 total

The last case pins the defect itself: with only err registered, { error: new Error('boom') } still serializes to {}. That test fails if someone removes the error entry.

npm run type-check clean. npx eslint src/utils/logger.ts src/startup/assertions.ts — 0 problems (the repo's 5 pre-existing lint errors are in other files).

End-to-end reproduction of the production crash

Built image run locally with every required variable set except RUVVECTOR_EMBEDDING_DIM — the exact environment of 00052-dxg.

Before (clean main) — reproduces the incident byte-for-byte, including the misleading "missing":0:

{"level":"info","present":5,"missing":0,"msg":"Environment variable assertion complete"}
{"level":"info","maxLatency":2000,"msg":"Performance budget configured"}
{"level":"fatal","error":{},"msg":"Failed to start server"}
exit=1

After (this branch):

{"level":"error","missing":["RUVVECTOR_EMBEDDING_DIM"],"isProduction":true,
 "msg":"STARTUP ASSERTION FAILED: Required environment variables missing"}
{"level":"fatal","error":{"type":"Error","message":"FATAL: Required environment variables missing:
 RUVVECTOR_EMBEDDING_DIM. Service cannot start safely without these values. DO NOT allow partial
 operation.","stack":"Error: ...\n    at assertRequiredEnvVars (dist/startup/assertions.js:92:15)\n
     at runStartupAssertions ...\n    at startServer ..."},"msg":"Failed to start server"}
exit=1

This confirms ADR-0003's root-cause hypothesis and closes its residual ~3% uncertainty (that assertValidDimension() was rejecting a malformed value instead). Both assertEmbeddingDimension() paths were also verified to log real errors now:

RUVVECTOR_EMBEDDING_DIM=0      → "FATAL: RUVVECTOR_EMBEDDING_DIM is required. ..."
RUVVECTOR_EMBEDDING_DIM=99999  → "Embedding dimension must be an integer between 1 and 16000, got: 99999"

⚠️ ACTION REQUIRED BEFORE THE NEXT DEPLOY — RUVVECTOR_EMBEDDING_DIM

This PR does NOT set RUVVECTOR_EMBEDDING_DIM anywhere. It is not in the code, not in a default, not in a config file, and not on any Cloud Run service. Merging this PR alone will not make the service start — it will make it tell you clearly why it didn't.

The variable must be set as a real deploy-time environment variable by a human. It deliberately has no default in code and must not be given one.

Candidate value: 1536 — requires explicit human confirmation before use.

• 1536 is the only candidate value evidenced anywhere in the workspace, and its single source is copilot-agent/.env.example:114 (EMBEDDING_DIMENSIONS=1536).
• That is an example file, not a confirmed producer configuration. It has not been checked against what copilot-agent and platform-agents/src/index.ts actually POST to /v1/vectors/store.
• This is a ONE-WAY DOOR. vector(N) fixes N in the table definition. VectorClient.connect() (src/clients/VectorClient.ts:151-157) refuses to start on any mismatch against the live column, and correcting a wrong choice after rows exist means a table rewrite plus a full backfill.
• ADR-0001's implementation plan item 1 states it directly: "Everything below is blocked on this number; do not guess it."

Because of that, 1536 must not be silently defaulted in code. Confirm the producer's true dimension first, then set it explicitly at deploy time:

RUVVECTOR_EMBEDDING_DIM=<confirmed> ./scripts/deploy-cloudrun.sh
# or, so all future revisions inherit it:
gcloud run services update ruvvector-service --region=us-central1 \
  --update-env-vars RUVVECTOR_EMBEDDING_DIM=<confirmed>

Note scripts/deploy-cloudrun.sh:18 still contains a plaintext database password (ADR-0002's scope, not fixed here) — remediate that before adopting the script as the standard deploy route.

Scope / safety

• Nothing deployed. Revision 00051-gwp is untouched and still serving 100% of traffic.
• No env var set on any live service.
• No change to runtime behavior, request handling, or the crash-on-misconfiguration posture. This changes what gets written down when the service crashes, and which variable the first assertion checks.
• Expect the next revision to surface a genuine database or schema error. Steps 2-4 of startServer() have never executed against this Cloud SQL instance with real persistence. That would be the logging fix working, not a regression.
• Log volume grows: full stack traces where {} used to be. Intended trade. Audited startup messages interpolate config but not credentials (EXECUTION_HMAC_SECRET is only ever logged by length) — worth re-checking whenever a new throw interpolates config.

Remaining ADR steps (open, not in this PR)

5. Confirm the embedding dimension — blocking, do not guess
6. Set the variable on the service — operator-approved action
7. Redeploy with --no-traffic, promote only once healthy
8. Address the follow-on failure, if any, using the now-real error message