feat(runtime-node): error-linked trace retention + Docker Desktop quickstart fixes

README.md

@@ -177,8 +177,10 @@ new OTLPTraceExporter({
 ## Design principles
 
 - **Errors are 100%, everything else is sampled or aggregated.** Raw error
-  occurrences are always kept (14-day TTL); successful traces are expected to
-  be sampled upstream (0.5–1%); usage is stored as 1-minute rollups (90 days).
+  occurrences are always kept (14-day TTL); traces containing an error are
+  retained in full (the Node SDK tail-retains them), so every issue keeps the
+  trace that explains it; healthy traces are expected to be sampled upstream
+  (0.5–1%); usage is stored as 1-minute rollups (90 days).
 - **Per-repo analysis.** Every row is keyed by `org_id` + `repository_id`.
 - **Privacy by construction.** No cookies, no DOM, no request/response bodies,
   no emails, no full URLs with query strings.

deploy/single-server/docker-compose.yml

@@ -15,7 +15,10 @@ services:
     # Not published to the host — only the ingester talks to it, over the
     # internal compose network. Never expose 8123/9000 publicly.
     healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "http://localhost:8123/ping"]
+      # 127.0.0.1, not localhost: in-container localhost resolves to ::1, but
+      # ClickHouse's IPv6 bind can silently fail (listen_try), leaving only
+      # 0.0.0.0 listening and the check refused forever.
+      test: ["CMD", "wget", "--spider", "-q", "http://127.0.0.1:8123/ping"]
       interval: 10s
       timeout: 5s
       retries: 10

docker-compose.yml

@@ -8,7 +8,10 @@ services:
     ulimits:
       nofile: 262144
     healthcheck:
-      test: ["CMD", "wget", "-qO-", "http://localhost:8123/ping"]
+      # 127.0.0.1, not localhost: in-container localhost resolves to ::1, but
+      # ClickHouse's IPv6 bind can silently fail (listen_try) — e.g. on Docker
+      # Desktop for Mac — leaving only 0.0.0.0 listening and the check refused.
+      test: ["CMD", "wget", "-qO-", "http://127.0.0.1:8123/ping"]
       interval: 5s
       retries: 10
 

docs/GETTING-STARTED.md

@@ -41,7 +41,7 @@ you from zero to seeing data in ClickHouse.
 | Unhandled errors / rejections | ✅ automatic | ✅ automatic |
 | Handled errors | `captureException(err)` | `captureException(err)` |
 | Usage | session pings + `trackEvent()` | request counts/durations per route (automatic) |
-| Traces | — (by design; no OTel in the browser) | ~1% sampled (configurable) |
+| Traces | — (by design; no OTel in the browser) | ~1% sampled, plus **every erroring trace kept in full** |
 | LLM usage & cost | — | `withLlmCall()` / Vercel AI SDK telemetry / GenAI semconv — always 100% (model, tokens, cost) |
 
 **What is never sent:** cookies, DOM content, form values, request/response
@@ -58,6 +58,11 @@ cd autter-runtime
 docker compose up   # ClickHouse + ingester on :4318, key "dev-key"
 ```
 
+> **That's all the clone is for.** It runs the ingester — you never add
+> code to this checkout. Every step from here on (installing packages,
+> creating `instrument.cjs`, …) happens in **your application's
+> repository**, the app you want to monitor.
+
 For real deployments, configure keys via env (or point
 `AUTTER_KEY_VALIDATOR_URL` at your own key service):
 
@@ -78,11 +83,15 @@ Full config reference: [`packages/otlp-ingester`](../packages/otlp-ingester).
 
 ## 3. Instrument your backend
 
+In **your app's repository** (not the `autter-runtime` checkout from
+step 2):
+
 ```bash
 npm install @autter/runtime-node
 ```
 
-Create `instrument.cjs` — it must load **before** your app:
+Create `instrument.cjs` in your app's root, next to its entry point — it
+must load **before** your app:
 
 ```js
 const { initAutterServer } = require("@autter/runtime-node");
@@ -101,8 +110,10 @@ node --require ./instrument.cjs server.js
 ```
 
 That alone gives you: every incoming HTTP request traced-and-sampled,
-request/error/duration rollups per route, and crashes captured. For
-handled errors:
+request/error/duration rollups per route, crashes captured, and the full
+trace of any request that errors — erroring traces are retained even when
+the sampler wouldn't have kept them, so every issue keeps the trace that
+explains it. For handled errors:
 
 ```js
 const { captureException } = require("@autter/runtime-node");
@@ -303,7 +314,9 @@ clickhouse-client --password dev`.
 - [ ] `release` is wired to your git SHA in **both** frontend and backend —
       it's what powers regression detection ("broke in release X").
 - [ ] Keep trace sampling at ~1% (`traceSampleRate`) — errors are always
-      captured regardless.
+      captured regardless, and the full trace of an erroring request is
+      retained (`retainTracesOnError`, on by default), so cheap sampling
+      doesn't cost you debugging context.
 - [ ] The relay route keeps its built-in per-IP rate limit (or your WAF
       covers it: `perIpRateLimit: false`).
 - [ ] Direct browser ingest: your CSP includes

docs/INTEGRATIONS.md

@@ -164,3 +164,15 @@ Errors are always worth sending; keep successful-trace sampling at ~1%
 (`OTEL_TRACES_SAMPLER_ARG=0.01`). Request/usage metrics are derived
 server-side from spans and the `http.server.duration` histogram — no extra
 setup. LLM/GenAI spans are the exception — send them at 100% (see above).
+
+One consequence of plain head sampling to know about: an error recorded on
+an unsampled trace is dropped with it, and even when the error survives
+through a separate path, the trace explaining it usually doesn't.
+`@autter/runtime-node` handles both automatically — errors ride a dedicated
+always-on tracer, and the full trace around an error is tail-retained even
+when head sampling said no (`retainTracesOnError`, on by default). On other
+stacks, route errors through an always-on tracer provider (same pattern as
+the LLM guidance above), and if you want erroring traces kept in full, put
+an OTel Collector with the `tail_sampling` processor (a
+`status_code = ERROR` policy plus a small `probabilistic` one) in front of
+the ingester instead of sampling in the SDK.

packages/otlp-ingester/src/normalize-otlp.ts

@@ -325,16 +325,12 @@
 				if (llmCall) llmCalls.push(llmCall);
 
 				// Server spans fold into 1-minute usage rollups so traffic is
-				// tracked even when the metrics pipeline isn't wired —
-				// senders that DO export request metrics mark the resource
-				// with `autter.metrics_wired` and are skipped here, or every
-				// request they trace would count twice. The rollup key uses
-				// the normalized route: `http.route` is already a template,
-				// but the url.path/http.target fallback is a raw path whose
-				// id segments would explode the SummingMergeTree key space
-				// (and never line up with the metric-fed rows for the same
-				// endpoint).
-				if (kind === "server" && !resource.metricsWired) {
+				// tracked even when the metrics pipeline isn't wired. Spans
+				// exported by error-linked tail retention are skipped: the SDK
+				// ships those at ~100% alongside a metrics pipeline that already
+				// counts every request, so folding them in would double-count
+				// erroring routes.
+				if (kind === "server" && attrs.get("autter.tail_retained") !== "true") {
 					addToRollup(rollups, {
 						service: resource.service,
 						environment: resource.environment,

packages/runtime-node/README.md

@@ -83,12 +83,23 @@ Defaults (cheap by construction):
 | Signal | Default |
 | --- | --- |
 | Captured/unhandled exceptions | 100% (dedicated always-on tracer) |
+| Traces containing an error | 100% (tail retention, `retainTracesOnError`) |
 | `withProcessSpan` spans | 100% (same always-on tracer) |
 | LLM/GenAI call spans | 100% (`llmTracing`, on by default) |
-| Traces | 1% head sampling (`traceSampleRate`) |
+| Healthy traces | 1% head sampling (`traceSampleRate`) |
 | Request metrics | exported every 60 s |
 | Logs | not collected |
 
+**Error-linked trace retention.** Errors export at 100% while traces are
+head-sampled — on its own that strands a retained error without the trace
+that explains it. So unsampled spans are kept briefly in an in-process
+buffer, and the moment a trace shows an error — a 5xx response, a recorded
+exception, `captureException`, or an error-severity `captureMessage` — the
+whole trace is exported, sampling lottery notwithstanding. The buffer is
+bounded (256 spans per trace, 5 000 spans total, dropped as soon as the
+request ends healthy, 30 s TTL), degrades to plain head sampling on
+overflow, and never blocks. Disable with `retainTracesOnError: false`.
+
 Crashes are observed via `process.uncaughtExceptionMonitor`, which does
 **not** change your process's exit behaviour; the final flush is
 best-effort. Framework instrumentations are opt-in:
@@ -102,6 +113,8 @@ Note on usage rollups: requests are counted from the `http.server.duration`
 metric (100% accurate) and additionally from sampled server spans. At the
 default 1% sampling the span contribution is negligible; if you set
 `traceSampleRate: 1` in development, expect request counts roughly doubled.
+Tail-retained error traces don't distort this: their spans carry
+`autter.tail_retained` and the ingester keeps them out of span-fed rollups.
 
 ## 3. LLM tracing