From 9da6800f0427dd09574f73a79d201d3d22ed57cd Mon Sep 17 00:00:00 2001 From: Rhys Sullivan Date: Fri, 12 Jun 2026 15:28:13 -0700 Subject: [PATCH 1/4] Tool failures become visible telemetry: outcome attributes, derived error messages, recording-tracer contracts MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Expected tool failures (ToolResult.fail) ride the Effect success channel, so the tracer recorded healthy spans while users hit upstream error walls — in a week of prod data, zero of 19.5k plugin.openapi.invoke spans carried http.status_code (annotated inside the inner OpenApi.invoke span, not the wrapper queries target) and 562 of 804 executor.tool.execute error spans had an empty status.message (tagged errors with no message field). - Stamp executor.tool.outcome / error_code / error_status on executor.tool.execute and mcp.tool.dispatch whenever a dispatch resolves to a ToolResult, plus tenant/subject attribution so queries no longer need a trace-id join against the outer request span. - Annotate http.status_code on the plugin.openapi.invoke wrapper span. - Derive messages for the message-less SDK tagged errors and stop formatInvocationCauseMessage rendering plain objects as [object Object]. - Add a recording Tracer to @executor-js/sdk/testing and telemetry contract tests that pin the span names, attributes, and error statuses production queries depend on. --- notes/product-observability-plan.md | 163 +++++++++++++ .../execution/src/telemetry-contract.test.ts | 220 ++++++++++++++++++ packages/core/execution/src/tool-invoker.ts | 5 + packages/core/sdk/src/elicitation.ts | 8 +- packages/core/sdk/src/errors.ts | 61 ++++- packages/core/sdk/src/executor.ts | 29 ++- packages/core/sdk/src/index.ts | 8 +- packages/core/sdk/src/testing.ts | 8 + .../core/sdk/src/testing/recording-tracer.ts | 80 +++++++ packages/core/sdk/src/tool-result.ts | 29 ++- packages/plugins/openapi/src/sdk/invoke.ts | 5 + .../openapi/src/sdk/upstream-failures.test.ts | 56 ++++- 12 files changed, 655 insertions(+), 17 deletions(-) create mode 100644 notes/product-observability-plan.md create mode 100644 packages/core/execution/src/telemetry-contract.test.ts create mode 100644 packages/core/sdk/src/testing/recording-tracer.ts diff --git a/notes/product-observability-plan.md b/notes/product-observability-plan.md new file mode 100644 index 000000000..d54d750be --- /dev/null +++ b/notes/product-observability-plan.md @@ -0,0 +1,163 @@ +# Product observability: know what's happening without watching + +Goal: errors surface to us before users report them; churn/configuration +health is understandable passively. Dogfood Executor for all of it — the +monitoring automations live in this repo, call our own integrations through +the Executor MCP, and exercise the "write automations in an executor folder +with a generated TS SDK" roadmap. + +## What exists today (audited 2026-06-12) + +**Signal that is already flowing:** + +- Worker traces → Axiom dataset `executor-cloud` (apps/cloud/src/observability/telemetry.ts). + Browser spans join via traceparent (#981/#985). +- Error spans DO exist: ~6.7k ERROR-status spans in the last 7d + (`http.server`, `executor.tool.execute` 804, `mcp.tool.dispatch` 211, + `plugin.mcp.*`, `plugin.openapi.invoke` 72, …). +- Sentry: cloud browser (with replay + tunnel), desktop (3 processes). + Worker-side capture only via explicit `captureCause` (no global init). +- PostHog: browser-only, cloud-only. The typed ~60-event product catalog is + built on branch `claude/jovial-panini-ee23b0` (not merged). + +**Proof the approach works — the SharePoint incident was in the data:** + +```apl +['executor-cloud'] +| where ['status.code'] == "ERROR" and name == "executor.tool.execute" +| extend msg = tostring(['status.message']) +``` + +shows `Missing required path parameter: drive-id/site-id/driveItem-id` on +`microsoft_graph…sitesGetDrives` / `sitesListDrives` / `drivesDriveItemSearch` +— 33 occurrences across 4 orgs, peaking 2026-06-09, days before the chat +where we debugged it. A daily digest would have caught it. Same window also +shows, unreported: `Stored refresh token could not be resolved.` ×75, +client-credentials token-exchange failures ×30, and `[object Object]` as a +status message ×22 (itself a bug — an error path that stringifies badly). + +**Org attribution works today via trace join** (org id lives on the outer +`mcp.request` span, not the tool span): + +```apl +['executor-cloud'] +| where name == "mcp.request" and isnotnull(['attributes.custom']['mcp.auth.organization_id']) +| project trace_id, org = tostring(['attributes.custom']['mcp.auth.organization_id']) +| join kind=inner ( + ['executor-cloud'] + | where ['status.code'] == "ERROR" and name == "executor.tool.execute" + | project trace_id, msg = substring(tostring(['status.message']),0,60) + ) on trace_id +| summarize n=count() by org, msg | sort by n desc +``` + +## Gaps (in priority order) + +1. **Expected tool failures are invisible to telemetry.** Upstream 4xx/5xx, + `connection_rejected`, `oauth_connection_missing` become + `ToolResult.fail(...)` in the Effect _success_ channel — no error span + status, no Sentry, no log. The `plugin.openapi.invoke` span gets + `http.status_code` stamped in code, but 0 of 19.5k spans in Axiom carry + it (attribute set after span end? verify). These are exactly the + "user is hitting a wall" signals. +2. **562/804 `executor.tool.execute` ERROR spans have an empty + status.message** — the biggest error class is unlabeled. +3. **No org/user/integration attributes on tool spans** — attribution + requires the trace join above; fine for digests, bad for Axiom monitors. +4. **No alerting exists anywhere.** No Axiom monitors, no scheduled checks, + nothing in CI. All of the above fires into a dataset nobody reads. +5. **No server-side product analytics.** PostHog is browser-only, so + MCP-driven usage (the actual product) is invisible to funnels/retention. + Server-event seams already mapped in the PostHog session + (ExecutionStackMiddleware, McpSessionStore.dispatch, the + `executor.tool.execute` span). + +## Plan + +### Layer 0 — verify the pipes (observability of the observability) + +The scariest failure mode is the signal silently going dark, in any of +these shapes: + +- errors handled "gracefully" and returned to the client without any + server-side record (the `ToolResult.fail` channel today); +- Sentry captures that never fire (worker has no global init — only + explicit `captureCause` callsites); +- spans exported but missing the attributes you'd query + (`http.status_code` stamped in code, present on 0/19.5k prod spans). + +None of these page anyone, because the absence of data looks identical to +health. The countermeasure is contract tests on the telemetry itself, so +regressions fail CI instead of being discovered during the next incident: + +- **Span contract tests (unit/integration):** drive a failing tool call + through the engine with an in-process span collector and assert the + exact spans + attributes + error statuses that Layer 1 promises. This is + what would have caught the status_code-after-span-end bug. +- **Motel-backed e2e:** the `E2E_MOTEL=1` path already exports + browser+server spans to local motel; add a scenario that triggers a tool + failure and asserts the trace in motel contains the error span with org + attribution. Pins the whole export pipeline, not just span creation. +- **Sentry emulator in `@executor-js/emulate`:** wire-level Sentry ingest + emulator (envelope endpoint + request ledger), point the worker/browser + DSN at it in e2e, assert "this user-visible error produced exactly one + Sentry event". Same pattern as the WorkOS emulator. +- **Prod canary (deploy-to-verify):** after the Layer-1 fixes deploy, run a + known-failing tool call against prod and confirm the error span lands in + Axiom with the expected attributes. Rhys has explicitly OK'd deploying + for this purpose (2026-06-12). A tiny scheduled "synthetic failure" canary + org can keep verifying the pipeline continuously — if the canary's error + spans stop appearing, THAT is the alert. + +Robustness here is what makes Layers 2–3 cheap: if the data is trustworthy +and complete, the monitoring on top is a handful of queries instead of an +ongoing forensic project. + +### Layer 1 — fix the telemetry at the source (small PRs, do first) + +- Mark tool-failure outcomes on spans: when `invokeTool` returns + `ToolResult.fail`, annotate the current span + (`executor.tool.outcome = fail`, `executor.tool.error_code`, + upstream status) and set error status for 5xx/auth-class failures. +- Fix empty + `[object Object]` status messages (normalize via the existing + `formatInvocationCauseMessage` path). +- Stamp `organization_id` / `account_id` / `mcp.tool.integration` onto + `executor.tool.execute` (values are already in AuthContext upstream). +- Merge the PostHog product-events branch; add the server-side event seam + behind the same no-op-by-default pattern. + +### Layer 2 — the `executor/` dogfood folder (the new thing) + +Create a top-level `executor/` directory: automations written against our +own integrations through the Executor MCP, exercising the roadmap +(folder of TS automations + generated SDK + schedules) on ourselves first. + +- `executor/automations/error-digest.ts` — daily: query Axiom for new/rising + error signatures (group by error-class × integration × org, diff vs + yesterday, call out first-seen signatures), post a digest. Would have + caught SharePoint, the refresh-token cluster, and `[object Object]`. +- `executor/automations/connection-health.ts` — failing-connection report: + orgs with repeated `oauth_refresh_failed` / `connection_rejected` (these + users silently churn). +- `executor/automations/usage-pulse.ts` — weekly: PostHog + Axiom joined + funnel/retention pulse (activation: spec added → connection working → + first successful tool call; orgs gone quiet). +- Generated typed SDK for the connected integrations (axiom, posthog, + planetscale, …) is the product feature this folder incubates; until the + generator exists, automations call the MCP execute surface directly. +- Scheduling: GH Actions cron (or scheduled cloud agent) with an Executor + API key; delivery channel TBD (Slack/email-via-Resend/GitHub issue). + +### Layer 3 — the Windmill-style in-product view + +Runs/activity console view: per-org recent invocations with outcome, +error class, drill-through to trace. The e2e runs-viewer work (#980) is the +in-house prior art; data needs are exactly the Layer-1 attributes. Design +after Layers 1–2 prove the queries. + +## Proven access paths + +- Axiom: `axiom_mcp.querydataset` (arg `apl`); error attrs live under + `['attributes.custom']`, status under `['status.code']`/`['status.message']`. +- PostHog: `posthog_api` (org key) + `mcp_posthog_com` OAuth both connected. +- Prod DB: `planetscale_mcp` (organization/database/branch args). diff --git a/packages/core/execution/src/telemetry-contract.test.ts b/packages/core/execution/src/telemetry-contract.test.ts new file mode 100644 index 000000000..60d0b6bb1 --- /dev/null +++ b/packages/core/execution/src/telemetry-contract.test.ts @@ -0,0 +1,220 @@ +// --------------------------------------------------------------------------- +// Telemetry contracts for the tool-dispatch path. +// +// The dangerous observability failure mode is the signal silently going dark: +// expected tool failures (`ToolResult.fail`) resolve through the Effect +// success channel, so without explicit outcome annotation the tracer records +// a healthy span for a user hitting an upstream error wall — absence of +// error data is indistinguishable from health. These tests drive real +// dispatches through a recording tracer and assert the spans, attributes, +// and error statuses that production queries (Axiom) depend on. A regression +// here fails CI instead of being discovered during the next incident. +// --------------------------------------------------------------------------- + +import { describe, expect, it } from "@effect/vitest"; +import { Data, Effect, Exit } from "effect"; + +import { + AuthTemplateSlug, + ConnectionName, + ConnectionNotFoundError, + ElicitationDeclinedError, + IntegrationSlug, + NoHandlerError, + PluginNotLoadedError, + ToolAddress, + ToolBlockedError, + ToolName, + ToolNotFoundError, + ToolResult, + createExecutor, + definePlugin, + type CredentialProvider, +} from "@executor-js/sdk"; +import { ProviderItemId, ProviderKey } from "@executor-js/sdk"; +import { + makeTestConfig, + runWithRecordingTracer, + spanEndedWithError, +} from "@executor-js/sdk/testing"; +import { makeExecutorToolInvoker } from "./tool-invoker"; + +const TEMPLATE = AuthTemplateSlug.make("apiKey"); +const CONN = ConnectionName.make("main"); +const INTEG = IntegrationSlug.make("upstream"); + +const memoryProvider = (): CredentialProvider => { + const store = new Map(); + return { + key: ProviderKey.make("telemetry-memory"), + writable: true, + get: (id) => Effect.sync(() => store.get(String(id)) ?? null), + set: (id, value) => Effect.sync(() => void store.set(String(id), value)), + list: () => + Effect.sync(() => + Array.from(store.keys()).map((key) => ({ + id: ProviderItemId.make(key), + name: key, + })), + ), + }; +}; + +const EmptyInputJson = { type: "object", properties: {} } as const; + +class TelemetryTestDefect extends Data.TaggedError("TelemetryTestDefect")<{ + readonly message: string; +}> {} + +// One integration, three tools spanning the outcome classes the telemetry +// contract distinguishes: domain success, expected upstream failure +// (success channel), and an infra defect (failure channel). +const telemetryPlugin = definePlugin(() => ({ + id: "telemetry-test" as const, + credentialProviders: [memoryProvider()], + storage: () => ({}), + resolveTools: () => + Effect.succeed({ + tools: [ + { name: ToolName.make("succeeds"), description: "", inputSchema: EmptyInputJson }, + { name: ToolName.make("failsUpstream"), description: "", inputSchema: EmptyInputJson }, + { name: ToolName.make("defects"), description: "", inputSchema: EmptyInputJson }, + ], + }), + invokeTool: ({ toolRow }) => { + if (toolRow.name === "succeeds") { + return Effect.succeed(ToolResult.ok({ fine: true })); + } + if (toolRow.name === "failsUpstream") { + return Effect.succeed( + ToolResult.fail({ + code: "upstream_http_error", + status: 502, + message: "Bad gateway from upstream", + }), + ); + } + return Effect.fail(new TelemetryTestDefect({ message: "database exploded" })); + }, + extension: (ctx) => ({ + seed: () => + ctx.core.integrations.register({ + slug: INTEG, + description: "telemetry upstream", + config: {}, + }), + }), +}))(); + +const makeHarness = () => + Effect.gen(function* () { + const executor = yield* createExecutor(makeTestConfig({ plugins: [telemetryPlugin] as const })); + yield* (executor as never as Record<"telemetry-test", { seed: () => Effect.Effect }>)[ + "telemetry-test" + ].seed(); + yield* executor.connections.create({ + owner: "org", + name: CONN, + integration: INTEG, + template: TEMPLATE, + value: "token", + }); + return makeExecutorToolInvoker(executor, { invokeOptions: {} }); + }); + +const dispatch = (tool: string) => + Effect.gen(function* () { + const invoker = yield* makeHarness(); + return yield* invoker.invoke({ path: `upstream.org.main.${tool}`, args: {} }); + }); + +describe("telemetry contract: tool dispatch spans", () => { + it.effect("a successful tool stamps outcome=ok on dispatch and execute spans", () => + Effect.gen(function* () { + const { exit, recording } = yield* runWithRecordingTracer(dispatch("succeeds")); + expect(Exit.isSuccess(exit)).toBe(true); + + const dispatchSpan = recording.single("mcp.tool.dispatch"); + expect(dispatchSpan.attributes.get("mcp.tool.name")).toBe("upstream.org.main.succeeds"); + expect(dispatchSpan.attributes.get("mcp.tool.integration")).toBe("upstream"); + expect(dispatchSpan.attributes.get("executor.tool.outcome")).toBe("ok"); + + const executeSpan = recording.single("executor.tool.execute"); + expect(executeSpan.attributes.get("executor.tool.outcome")).toBe("ok"); + // Org/tenant attribution on the tool span itself — production queries + // must not need a trace-id join against the outer request span. + expect(executeSpan.attributes.get("executor.tenant")).toBe("test-tenant"); + expect(executeSpan.attributes.get("executor.subject")).toBe("test-subject"); + }), + ); + + it.effect( + "an expected upstream failure (success channel) stamps outcome=fail + code + status", + () => + Effect.gen(function* () { + const { exit, recording } = yield* runWithRecordingTracer(dispatch("failsUpstream")); + // The contract under test: the failure is a VALUE, not an Effect error… + expect(Exit.isSuccess(exit)).toBe(true); + + // …so the span must carry the outcome explicitly, on both the + // sandbox-dispatch span and the executor-execute span. + for (const name of ["mcp.tool.dispatch", "executor.tool.execute"]) { + const span = recording.single(name); + expect(span.attributes.get("executor.tool.outcome")).toBe("fail"); + expect(span.attributes.get("executor.tool.error_code")).toBe("upstream_http_error"); + expect(span.attributes.get("executor.tool.error_status")).toBe(502); + } + }), + ); + + it.effect("an infra defect ends the dispatch span with an error exit", () => + Effect.gen(function* () { + const { exit, recording } = yield* runWithRecordingTracer(dispatch("defects")); + expect(Exit.isFailure(exit)).toBe(true); + + const dispatchSpan = recording.single("mcp.tool.dispatch"); + expect(spanEndedWithError(dispatchSpan)).toBe(true); + }), + ); + + it.effect("tool_not_found surfaces as outcome=fail with its code on the dispatch span", () => + Effect.gen(function* () { + const { exit, recording } = yield* runWithRecordingTracer(dispatch("doesNotExist")); + // tool_not_found is an expected failure: surfaced as a ToolResult.fail + // VALUE through the success channel. + expect(Exit.isSuccess(exit)).toBe(true); + + const dispatchSpan = recording.single("mcp.tool.dispatch"); + expect(dispatchSpan.attributes.get("executor.tool.outcome")).toBe("fail"); + expect(dispatchSpan.attributes.get("executor.tool.error_code")).toBe("tool_not_found"); + }), + ); +}); + +describe("telemetry contract: error messages", () => { + it("sdk tagged errors render a non-empty message for span status", () => { + // These messages become OTLP status.message via Cause.prettyErrors — + // before the derived getters, 562 of 804 error spans in a week of prod + // data had an EMPTY status message (the error class defined no + // `message`, and TaggedErrorClass instances default to ""). + const address = ToolAddress.make("tools.upstream.org.main.failsUpstream"); + const errors: ReadonlyArray = [ + new ToolNotFoundError({ address }), + new ToolBlockedError({ address, pattern: "*" }), + new PluginNotLoadedError({ address, pluginId: "p" }), + new NoHandlerError({ address, pluginId: "p" }), + new ConnectionNotFoundError({ + owner: "org", + integration: INTEG, + name: CONN, + }), + new ElicitationDeclinedError({ address, action: "decline" }), + ]; + for (const error of errors) { + // oxlint-disable-next-line executor/no-unknown-error-message -- the test asserts ON the message contract itself + const { message, name } = error; + expect(message.length, `${name} must derive a message`).toBeGreaterThan(0); + expect(message).not.toContain("[object Object]"); + } + }); +}); diff --git a/packages/core/execution/src/tool-invoker.ts b/packages/core/execution/src/tool-invoker.ts index 619170b61..4e3448095 100644 --- a/packages/core/execution/src/tool-invoker.ts +++ b/packages/core/execution/src/tool-invoker.ts @@ -9,6 +9,7 @@ import type { ToolSchemaView, } from "@executor-js/sdk/core"; import { + annotateToolResultOutcome, authToolFailure, isToolResult, ToolResult, @@ -282,6 +283,10 @@ export const makeExecutorToolInvoker = ( // raw success value and wrapped — keeps the sandbox-facing contract // uniform without forcing every tiny test plugin to import // `ToolResult.ok`. + // Expected failures resolve through the success channel, so without the + // outcome annotation the dispatch span reads as healthy even when the + // caller hit an upstream error or auth wall. + yield* annotateToolResultOutcome(result); if (isToolResult(result)) { return result; } diff --git a/packages/core/sdk/src/elicitation.ts b/packages/core/sdk/src/elicitation.ts index 87b409698..213c9c623 100644 --- a/packages/core/sdk/src/elicitation.ts +++ b/packages/core/sdk/src/elicitation.ts @@ -62,4 +62,10 @@ export class ElicitationDeclinedError extends Schema.TaggedErrorClass()( "ToolNotFoundError", { address: ToolAddress, suggestions: Schema.optional(Schema.Array(ToolAddress)), }, -) {} +) { + override get message(): string { + return `Tool not found: ${this.address}`; + } +} export class ToolInvocationError extends Schema.TaggedErrorClass()( "ToolInvocationError", @@ -38,7 +49,11 @@ export class ToolBlockedError extends Schema.TaggedErrorClass( address: ToolAddress, pattern: Schema.String, }, -) {} +) { + override get message(): string { + return `Tool blocked by policy "${this.pattern}": ${this.address}`; + } +} /** Tool row exists but its owning plugin isn't loaded in this executor config. */ export class PluginNotLoadedError extends Schema.TaggedErrorClass()( @@ -47,13 +62,21 @@ export class PluginNotLoadedError extends Schema.TaggedErrorClass()("NoHandlerError", { address: ToolAddress, pluginId: Schema.String, -}) {} +}) { + override get message(): string { + return `Plugin "${this.pluginId}" has no invokeTool handler for tool: ${this.address}`; + } +} // --------------------------------------------------------------------------- // Integration / connection lifecycle @@ -62,7 +85,11 @@ export class NoHandlerError extends Schema.TaggedErrorClass()("N export class IntegrationNotFoundError extends Schema.TaggedErrorClass()( "IntegrationNotFoundError", { slug: IntegrationSlug }, -) {} +) { + override get message(): string { + return `Integration not found: ${this.slug}`; + } +} /** An "add integration" operation targeted a slug (namespace) that is already * registered. The core `integrations.register` primitive upserts by design @@ -73,14 +100,22 @@ export class IntegrationAlreadyExistsError extends Schema.TaggedErrorClass()( "IntegrationRemovalNotAllowedError", { slug: IntegrationSlug }, -) {} +) { + override get message(): string { + return `Integration cannot be removed (declared statically by a plugin): ${this.slug}`; + } +} export class ConnectionNotFoundError extends Schema.TaggedErrorClass()( "ConnectionNotFoundError", @@ -89,7 +124,11 @@ export class ConnectionNotFoundError extends Schema.TaggedErrorClass()( "CredentialProviderNotRegisteredError", { provider: ProviderKey }, -) {} +) { + override get message(): string { + return `Credential provider not registered: ${this.provider}`; + } +} /** A connection's value could not be resolved — the provider returned nothing, * or an OAuth token refresh failed and the user must re-auth. */ diff --git a/packages/core/sdk/src/executor.ts b/packages/core/sdk/src/executor.ts index c8d578bf8..2413d5549 100644 --- a/packages/core/sdk/src/executor.ts +++ b/packages/core/sdk/src/executor.ts @@ -1,4 +1,4 @@ -import { Effect, Layer, Option, Predicate, Schema } from "effect"; +import { Effect, Inspectable, Layer, Option, Predicate, Schema } from "effect"; import { FetchHttpClient, type HttpClient } from "effect/unstable/http"; import { fumadb } from "@executor-js/fumadb"; import { memoryAdapter } from "@executor-js/fumadb/adapters/memory"; @@ -139,6 +139,7 @@ import { type OAuthEndpointUrlPolicy, } from "./oauth-helpers"; import { connectionIdentifier } from "./connection-name-identifier"; +import { annotateToolResultOutcome } from "./tool-result"; const PLUGIN_STORAGE_DELETE_KEY_BATCH_SIZE = 90; const MAX_APPROVAL_ARGUMENT_PREVIEW_CHARS = 4_000; @@ -2719,10 +2720,20 @@ export const createExecutor = => { const handler = pickHandler(options); return Effect.gen(function* () { + // oxlint-disable executor/no-instanceof-error, executor/no-unknown-error-message, executor/no-manual-tag-check -- boundary: normalize arbitrary unknown plugin failures into a human-readable message for ToolInvocationError/telemetry const formatInvocationCauseMessage = (cause: unknown): string => { - // oxlint-disable-next-line executor/no-instanceof-error, executor/no-unknown-error-message -- boundary: preserve public execute error message wrapping for unknown plugin failures - return cause instanceof Error ? cause.message : String(cause); + if (cause instanceof Error && cause.message.length > 0) return cause.message; + // Non-Error / empty-message causes: `String(plainObject)` renders + // "[object Object]", which is what telemetry then shows as the only + // label for the failure. Prefer the tag, else stringify structurally. + if (typeof cause === "object" && cause !== null) { + const tag = (cause as { readonly _tag?: unknown })._tag; + if (typeof tag === "string") return tag; + return Inspectable.toStringUnknown(cause, 0); + } + return String(cause); }; + // oxlint-enable executor/no-instanceof-error, executor/no-unknown-error-message, executor/no-manual-tag-check const wrapInvocationError = ( effect: Effect.Effect, ): Effect.Effect => @@ -2877,8 +2888,18 @@ export const createExecutor = ; +} + +export interface RecordingTracer { + readonly tracer: Tracer.Tracer; + readonly spans: readonly RecordedSpan[]; + /** All recorded spans with the given name. */ + readonly byName: (name: string) => readonly RecordedSpan[]; + /** Exactly-one convenience: throws when zero or multiple spans match. */ + readonly single: (name: string) => RecordedSpan; +} + +export const makeRecordingTracer = (): RecordingTracer => { + const spans: RecordedSpan[] = []; + const tracer = Tracer.make({ + span: (options) => { + const span = new Tracer.NativeSpan(options); + spans.push({ name: options.name, span, attributes: span.attributes }); + return span; + }, + }); + return { + tracer, + spans, + byName: (name) => spans.filter((entry) => entry.name === name), + single: (name) => { + const matches = spans.filter((entry) => entry.name === name); + if (matches.length !== 1) { + // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-error-constructor -- test helper: a wrong span count is a test bug, fail the test loud + throw new Error( + `Expected exactly one "${name}" span, recorded ${matches.length} (all spans: ${spans.map((entry) => entry.name).join(", ")})`, + ); + } + return matches[0]!; + }, + }; +}; + +/** Run an effect with a recording tracer and return both its exit and the + * recorded spans. The effect's failure is captured, not thrown — telemetry + * contracts usually assert on spans of FAILING operations. */ +export const runWithRecordingTracer = ( + effect: Effect.Effect, +): Effect.Effect<{ + readonly exit: Exit.Exit; + readonly recording: RecordingTracer; +}> => + Effect.suspend(() => { + const recording = makeRecordingTracer(); + return Effect.exit(effect).pipe( + Effect.withTracer(recording.tracer), + Effect.map((exit) => ({ exit, recording })), + ); + }); + +/** The span's ended exit, or null while started. */ +export const spanExit = (entry: RecordedSpan): Exit.Exit | null => { + const status = entry.span.status; + return Predicate.isTagged(status, "Ended") ? status.exit : null; +}; + +/** True when the span ended with a failure exit (what OTLP exporters map to + * status ERROR). */ +export const spanEndedWithError = (entry: RecordedSpan): boolean => { + const exit = spanExit(entry); + return exit != null && Exit.isFailure(exit); +}; diff --git a/packages/core/sdk/src/tool-result.ts b/packages/core/sdk/src/tool-result.ts index 389ec3a6b..24334d8ed 100644 --- a/packages/core/sdk/src/tool-result.ts +++ b/packages/core/sdk/src/tool-result.ts @@ -5,7 +5,7 @@ // the Effect failure channel. // --------------------------------------------------------------------------- -import { Schema } from "effect"; +import { Effect, Schema } from "effect"; export const ToolErrorSchema = Schema.Struct({ code: Schema.String, @@ -56,3 +56,30 @@ const isUnknownToolResult = Schema.is(ToolResultSchema); export const isToolResult = (value: unknown): value is ToolResult => isUnknownToolResult(value); + +/** + * Annotate the current span with the outcome of a tool invocation. + * + * `ToolResult.fail` rides the Effect *success* channel by design (expected + * failures are values, not defects), which means the tracer records those + * spans as healthy. Without this, "user keeps hitting 4xx walls" is invisible + * to telemetry — the exact class of signal that lets us catch product issues + * before they're reported. Stamped attributes: + * + * - `executor.tool.outcome` — "ok" | "fail" (always, on ToolResults) + * - `executor.tool.error_code` — ToolError.code (fail only) + * - `executor.tool.error_status` — upstream HTTP status (fail, when present) + * + * Codes/statuses are enumerable identifiers, never user content — safe span + * attributes. Non-ToolResult values (raw success payloads) annotate "ok". + */ +export const annotateToolResultOutcome = (value: unknown): Effect.Effect => { + if (isToolResult(value) && !value.ok) { + return Effect.annotateCurrentSpan({ + "executor.tool.outcome": "fail", + "executor.tool.error_code": value.error.code, + ...(value.error.status != null ? { "executor.tool.error_status": value.error.status } : {}), + }); + } + return Effect.annotateCurrentSpan({ "executor.tool.outcome": "ok" }); +}; diff --git a/packages/plugins/openapi/src/sdk/invoke.ts b/packages/plugins/openapi/src/sdk/invoke.ts index df36cf13e..48fe06bf7 100644 --- a/packages/plugins/openapi/src/sdk/invoke.ts +++ b/packages/plugins/openapi/src/sdk/invoke.ts @@ -625,6 +625,11 @@ export const invokeWithLayer = ( return invoke(operation, args, resolvedHeaders, sourceQueryParams).pipe( Effect.provide(clientWithBaseUrl), + // `invoke` annotates http.status_code on ITS span (`OpenApi.invoke`, + // via Effect.fn) — annotateCurrentSpan inside it never reaches this + // wrapper span. Stamp the status here too so queries against + // `plugin.openapi.invoke` see the upstream outcome directly. + Effect.tap((result) => Effect.annotateCurrentSpan({ "http.status_code": result.status })), Effect.withSpan("plugin.openapi.invoke", { attributes: { "plugin.openapi.method": operation.method.toUpperCase(), diff --git a/packages/plugins/openapi/src/sdk/upstream-failures.test.ts b/packages/plugins/openapi/src/sdk/upstream-failures.test.ts index ba2389423..42e2be694 100644 --- a/packages/plugins/openapi/src/sdk/upstream-failures.test.ts +++ b/packages/plugins/openapi/src/sdk/upstream-failures.test.ts @@ -32,7 +32,11 @@ import { IntegrationSlug, ToolAddress, } from "@executor-js/sdk"; -import { makeTestConfig, memoryCredentialsPlugin } from "@executor-js/sdk/testing"; +import { + makeTestConfig, + memoryCredentialsPlugin, + runWithRecordingTracer, +} from "@executor-js/sdk/testing"; import { addOpenApiTestConnection, makeOpenApiHttpApiTestSourceConfig, @@ -268,6 +272,56 @@ describe("OpenAPI upstream failure modes", () => { }), ); + // Telemetry contract: the spans production queries depend on must carry + // the upstream outcome. This pins two real prod regressions: (1) + // `http.status_code` was annotated inside `OpenApi.invoke` but queries + // target the `plugin.openapi.invoke` wrapper span, which carried it on + // 0 of 19.5k spans; (2) ToolResult.fail rides the success channel, so + // without outcome attributes a 5xx-spewing integration looks healthy. + it.effect("a failing invocation stamps status + outcome on its spans", () => + Effect.gen(function* () { + const server = yield* startScriptedServer(() => ({ + status: 502, + headers: { "content-type": "application/json" }, + body: '{"error":{"message":"bad gateway"}}', + })); + const { executor, address } = yield* buildExecutorForOpenApiServer(server); + + const { exit, recording } = yield* runWithRecordingTracer(executor.execute(address, {})); + expect(Exit.isSuccess(exit)).toBe(true); + + const invokeSpan = recording.single("plugin.openapi.invoke"); + expect(invokeSpan.attributes.get("http.status_code")).toBe(502); + expect(invokeSpan.attributes.get("plugin.openapi.method")).toBe("GET"); + + const executeSpan = recording.single("executor.tool.execute"); + expect(executeSpan.attributes.get("executor.tool.outcome")).toBe("fail"); + expect(executeSpan.attributes.get("executor.tool.error_code")).toBe("upstream_http_error"); + expect(executeSpan.attributes.get("executor.tool.error_status")).toBe(502); + expect(executeSpan.attributes.get("executor.tenant")).toBe("test-tenant"); + }), + ); + + it.effect("a successful invocation stamps status + outcome=ok on its spans", () => + Effect.gen(function* () { + const server = yield* startScriptedServer(() => ({ + status: 200, + headers: { "content-type": "application/json" }, + body: "[]", + })); + const { executor, address } = yield* buildExecutorForOpenApiServer(server); + + const { exit, recording } = yield* runWithRecordingTracer(executor.execute(address, {})); + expect(Exit.isSuccess(exit)).toBe(true); + + const invokeSpan = recording.single("plugin.openapi.invoke"); + expect(invokeSpan.attributes.get("http.status_code")).toBe(200); + + const executeSpan = recording.single("executor.tool.execute"); + expect(executeSpan.attributes.get("executor.tool.outcome")).toBe("ok"); + }), + ); + it.effect("upstream slow-then-respond doesn't lose the request", () => Effect.gen(function* () { const slowServer = Effect.acquireRelease( From e7c6149905150326a8a23a83db9bba988b925e1d Mon Sep 17 00:00:00 2001 From: Rhys Sullivan Date: Fri, 12 Jun 2026 15:43:11 -0700 Subject: [PATCH 2/4] Move product-observability plan note out of the repo --- notes/product-observability-plan.md | 163 ---------------------------- 1 file changed, 163 deletions(-) delete mode 100644 notes/product-observability-plan.md diff --git a/notes/product-observability-plan.md b/notes/product-observability-plan.md deleted file mode 100644 index d54d750be..000000000 --- a/notes/product-observability-plan.md +++ /dev/null @@ -1,163 +0,0 @@ -# Product observability: know what's happening without watching - -Goal: errors surface to us before users report them; churn/configuration -health is understandable passively. Dogfood Executor for all of it — the -monitoring automations live in this repo, call our own integrations through -the Executor MCP, and exercise the "write automations in an executor folder -with a generated TS SDK" roadmap. - -## What exists today (audited 2026-06-12) - -**Signal that is already flowing:** - -- Worker traces → Axiom dataset `executor-cloud` (apps/cloud/src/observability/telemetry.ts). - Browser spans join via traceparent (#981/#985). -- Error spans DO exist: ~6.7k ERROR-status spans in the last 7d - (`http.server`, `executor.tool.execute` 804, `mcp.tool.dispatch` 211, - `plugin.mcp.*`, `plugin.openapi.invoke` 72, …). -- Sentry: cloud browser (with replay + tunnel), desktop (3 processes). - Worker-side capture only via explicit `captureCause` (no global init). -- PostHog: browser-only, cloud-only. The typed ~60-event product catalog is - built on branch `claude/jovial-panini-ee23b0` (not merged). - -**Proof the approach works — the SharePoint incident was in the data:** - -```apl -['executor-cloud'] -| where ['status.code'] == "ERROR" and name == "executor.tool.execute" -| extend msg = tostring(['status.message']) -``` - -shows `Missing required path parameter: drive-id/site-id/driveItem-id` on -`microsoft_graph…sitesGetDrives` / `sitesListDrives` / `drivesDriveItemSearch` -— 33 occurrences across 4 orgs, peaking 2026-06-09, days before the chat -where we debugged it. A daily digest would have caught it. Same window also -shows, unreported: `Stored refresh token could not be resolved.` ×75, -client-credentials token-exchange failures ×30, and `[object Object]` as a -status message ×22 (itself a bug — an error path that stringifies badly). - -**Org attribution works today via trace join** (org id lives on the outer -`mcp.request` span, not the tool span): - -```apl -['executor-cloud'] -| where name == "mcp.request" and isnotnull(['attributes.custom']['mcp.auth.organization_id']) -| project trace_id, org = tostring(['attributes.custom']['mcp.auth.organization_id']) -| join kind=inner ( - ['executor-cloud'] - | where ['status.code'] == "ERROR" and name == "executor.tool.execute" - | project trace_id, msg = substring(tostring(['status.message']),0,60) - ) on trace_id -| summarize n=count() by org, msg | sort by n desc -``` - -## Gaps (in priority order) - -1. **Expected tool failures are invisible to telemetry.** Upstream 4xx/5xx, - `connection_rejected`, `oauth_connection_missing` become - `ToolResult.fail(...)` in the Effect _success_ channel — no error span - status, no Sentry, no log. The `plugin.openapi.invoke` span gets - `http.status_code` stamped in code, but 0 of 19.5k spans in Axiom carry - it (attribute set after span end? verify). These are exactly the - "user is hitting a wall" signals. -2. **562/804 `executor.tool.execute` ERROR spans have an empty - status.message** — the biggest error class is unlabeled. -3. **No org/user/integration attributes on tool spans** — attribution - requires the trace join above; fine for digests, bad for Axiom monitors. -4. **No alerting exists anywhere.** No Axiom monitors, no scheduled checks, - nothing in CI. All of the above fires into a dataset nobody reads. -5. **No server-side product analytics.** PostHog is browser-only, so - MCP-driven usage (the actual product) is invisible to funnels/retention. - Server-event seams already mapped in the PostHog session - (ExecutionStackMiddleware, McpSessionStore.dispatch, the - `executor.tool.execute` span). - -## Plan - -### Layer 0 — verify the pipes (observability of the observability) - -The scariest failure mode is the signal silently going dark, in any of -these shapes: - -- errors handled "gracefully" and returned to the client without any - server-side record (the `ToolResult.fail` channel today); -- Sentry captures that never fire (worker has no global init — only - explicit `captureCause` callsites); -- spans exported but missing the attributes you'd query - (`http.status_code` stamped in code, present on 0/19.5k prod spans). - -None of these page anyone, because the absence of data looks identical to -health. The countermeasure is contract tests on the telemetry itself, so -regressions fail CI instead of being discovered during the next incident: - -- **Span contract tests (unit/integration):** drive a failing tool call - through the engine with an in-process span collector and assert the - exact spans + attributes + error statuses that Layer 1 promises. This is - what would have caught the status_code-after-span-end bug. -- **Motel-backed e2e:** the `E2E_MOTEL=1` path already exports - browser+server spans to local motel; add a scenario that triggers a tool - failure and asserts the trace in motel contains the error span with org - attribution. Pins the whole export pipeline, not just span creation. -- **Sentry emulator in `@executor-js/emulate`:** wire-level Sentry ingest - emulator (envelope endpoint + request ledger), point the worker/browser - DSN at it in e2e, assert "this user-visible error produced exactly one - Sentry event". Same pattern as the WorkOS emulator. -- **Prod canary (deploy-to-verify):** after the Layer-1 fixes deploy, run a - known-failing tool call against prod and confirm the error span lands in - Axiom with the expected attributes. Rhys has explicitly OK'd deploying - for this purpose (2026-06-12). A tiny scheduled "synthetic failure" canary - org can keep verifying the pipeline continuously — if the canary's error - spans stop appearing, THAT is the alert. - -Robustness here is what makes Layers 2–3 cheap: if the data is trustworthy -and complete, the monitoring on top is a handful of queries instead of an -ongoing forensic project. - -### Layer 1 — fix the telemetry at the source (small PRs, do first) - -- Mark tool-failure outcomes on spans: when `invokeTool` returns - `ToolResult.fail`, annotate the current span - (`executor.tool.outcome = fail`, `executor.tool.error_code`, - upstream status) and set error status for 5xx/auth-class failures. -- Fix empty + `[object Object]` status messages (normalize via the existing - `formatInvocationCauseMessage` path). -- Stamp `organization_id` / `account_id` / `mcp.tool.integration` onto - `executor.tool.execute` (values are already in AuthContext upstream). -- Merge the PostHog product-events branch; add the server-side event seam - behind the same no-op-by-default pattern. - -### Layer 2 — the `executor/` dogfood folder (the new thing) - -Create a top-level `executor/` directory: automations written against our -own integrations through the Executor MCP, exercising the roadmap -(folder of TS automations + generated SDK + schedules) on ourselves first. - -- `executor/automations/error-digest.ts` — daily: query Axiom for new/rising - error signatures (group by error-class × integration × org, diff vs - yesterday, call out first-seen signatures), post a digest. Would have - caught SharePoint, the refresh-token cluster, and `[object Object]`. -- `executor/automations/connection-health.ts` — failing-connection report: - orgs with repeated `oauth_refresh_failed` / `connection_rejected` (these - users silently churn). -- `executor/automations/usage-pulse.ts` — weekly: PostHog + Axiom joined - funnel/retention pulse (activation: spec added → connection working → - first successful tool call; orgs gone quiet). -- Generated typed SDK for the connected integrations (axiom, posthog, - planetscale, …) is the product feature this folder incubates; until the - generator exists, automations call the MCP execute surface directly. -- Scheduling: GH Actions cron (or scheduled cloud agent) with an Executor - API key; delivery channel TBD (Slack/email-via-Resend/GitHub issue). - -### Layer 3 — the Windmill-style in-product view - -Runs/activity console view: per-org recent invocations with outcome, -error class, drill-through to trace. The e2e runs-viewer work (#980) is the -in-house prior art; data needs are exactly the Layer-1 attributes. Design -after Layers 1–2 prove the queries. - -## Proven access paths - -- Axiom: `axiom_mcp.querydataset` (arg `apl`); error attrs live under - `['attributes.custom']`, status under `['status.code']`/`['status.message']`. -- PostHog: `posthog_api` (org key) + `mcp_posthog_com` OAuth both connected. -- Prod DB: `planetscale_mcp` (organization/database/branch args). From 7b3338f720fdf09b9b1e34f8bd9888e2add1c6ca Mon Sep 17 00:00:00 2001 From: Rhys Sullivan Date: Fri, 12 Jun 2026 16:08:25 -0700 Subject: [PATCH 3/4] Pin the telemetry contract end-to-end instead of unit-level MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the recording-tracer unit tests with one e2e scenario that drives the whole production topology — HTTP API → execution engine → sandbox → OpenAPI invoke → a real upstream returning 502 → OTLP export — and asserts the spans as the suite's motel store actually received them: outcome/error_code/error_status + tenant on executor.tool.execute, http.status_code on plugin.openapi.invoke, and ok/fail distinguishable. The unit tests verified span objects in process; the prod bug class this guards against (attribute stamped on a span the exporter never carries, failure riding the success channel) only shows up at the exported layer. Adds a Telemetry e2e service (motel span search with arrival polling), provided when the suite's motel boots (E2E_MOTEL_URL). --- e2e/cloud/telemetry-contract.test.ts | 181 ++++++++++++++ e2e/setup/cloud.globalsetup.ts | 4 + e2e/src/scenario.ts | 8 +- e2e/src/services.ts | 5 + e2e/src/surfaces/telemetry.ts | 75 ++++++ .../execution/src/telemetry-contract.test.ts | 220 ------------------ packages/core/sdk/src/testing.ts | 8 - .../core/sdk/src/testing/recording-tracer.ts | 80 ------- .../openapi/src/sdk/upstream-failures.test.ts | 56 +---- 9 files changed, 273 insertions(+), 364 deletions(-) create mode 100644 e2e/cloud/telemetry-contract.test.ts create mode 100644 e2e/src/surfaces/telemetry.ts delete mode 100644 packages/core/execution/src/telemetry-contract.test.ts delete mode 100644 packages/core/sdk/src/testing/recording-tracer.ts diff --git a/e2e/cloud/telemetry-contract.test.ts b/e2e/cloud/telemetry-contract.test.ts new file mode 100644 index 000000000..680c15b6d --- /dev/null +++ b/e2e/cloud/telemetry-contract.test.ts @@ -0,0 +1,181 @@ +// Cloud: the telemetry contract, end to end. A tool call that hits an +// upstream error wall must be visible in the EXPORTED spans — not just +// handled gracefully for the caller. This is the regression class where the +// product silently goes dark to operators: `ToolResult.fail` rides the +// Effect success channel (a healthy-looking span), and an attribute stamped +// on the wrong span simply never arrives in the trace store, which looks +// identical to health. So the assertion runs against the OTLP store the dev +// stack actually exported to (the suite's motel — the same exporter layer +// that ships prod spans to Axiom), driving the whole production topology: +// HTTP API → execution engine → sandbox → OpenAPI invoke → a real upstream +// returning 502 → span batch → OTLP export. +// +// Pins two regressions found live in prod (2026-06-12): http.status_code was +// annotated inside the inner `OpenApi.invoke` span so the `plugin.openapi. +// invoke` span queries target carried it on 0 of ~19.5k spans; and failed +// tool calls were indistinguishable from successes on `executor.tool.execute`. +import { randomBytes } from "node:crypto"; +import { createServer } from "node:http"; + +import { expect } from "@effect/vitest"; +import { Effect } from "effect"; +import { composePluginApi } from "@executor-js/api/server"; +import { openApiHttpPlugin } from "@executor-js/plugin-openapi/api"; +import { AuthTemplateSlug, ConnectionName, IntegrationSlug } from "@executor-js/sdk/shared"; + +import { scenario } from "../src/scenario"; +import { Api, Target, Telemetry } from "../src/services"; + +const api = composePluginApi([openApiHttpPlugin()] as const); + +/** Two operations: /ok answers 200, /fail answers 502 — the success and + * expected-upstream-failure outcome classes the telemetry must separate. */ +const upstreamSpec = (baseUrl: string): string => + JSON.stringify({ + openapi: "3.0.3", + info: { title: "Telemetry Upstream", version: "1.0.0" }, + servers: [{ url: baseUrl }], + paths: { + "/ok": { + get: { + operationId: "ok", + summary: "Succeeds", + tags: ["probe"], + responses: { "200": { description: "" } }, + }, + }, + "/fail": { + get: { + operationId: "fail", + summary: "Always 502", + tags: ["probe"], + responses: { "200": { description: "" } }, + }, + }, + }, + }); + +/** A real upstream on 127.0.0.1: /ok → 200 JSON, anything else → 502 JSON. */ +const serveUpstream = Effect.acquireRelease( + Effect.callback<{ readonly baseUrl: string; readonly close: () => void }>((resume) => { + const server = createServer((request, response) => { + const ok = request.url?.startsWith("/ok") ?? false; + response.writeHead(ok ? 200 : 502, { "content-type": "application/json" }); + response.end(ok ? '{"fine":true}' : '{"error":{"message":"bad gateway"}}'); + }); + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + const port = typeof address === "object" && address ? address.port : 0; + resume( + Effect.succeed({ + baseUrl: `http://127.0.0.1:${port}`, + close: () => { + server.close(); + server.closeAllConnections(); + }, + }), + ); + }); + }), + (server) => Effect.sync(server.close), +); + +scenario( + "Telemetry · a failing tool call is visible in the exported spans", + { timeout: 180_000 }, + Effect.scoped( + Effect.gen(function* () { + const target = yield* Target; + const { client: apiClient } = yield* Api; + const telemetry = yield* Telemetry; + const identity = yield* target.newIdentity(); + const client = yield* apiClient(api, identity); + + const upstream = yield* serveUpstream; + + // Identifier-safe slug: it becomes a property path in the sandbox code. + const slug = IntegrationSlug.make(`telscn${randomBytes(4).toString("hex")}`); + yield* client.openapi.addSpec({ + payload: { + spec: { kind: "blob", value: upstreamSpec(upstream.baseUrl) }, + slug, + baseUrl: upstream.baseUrl, + authenticationTemplate: [ + { + slug: "apiKey", + type: "apiKey", + headers: { Authorization: ["Bearer ", { type: "variable", name: "token" }] }, + }, + ], + }, + }); + yield* client.connections.create({ + payload: { + owner: "org", + name: ConnectionName.make("main"), + integration: slug, + template: AuthTemplateSlug.make("apiKey"), + value: "telemetry-scenario-token", + }, + }); + + const tools = yield* client.tools.list({ query: {} }); + const addressOf = (op: string) => { + const tool = tools.find( + (entry) => + String(entry.integration) === String(slug) && String(entry.address).endsWith(`.${op}`), + ); + expect(tool, `the ${op} tool is in the catalog`).toBeDefined(); + return String(tool!.address); + }; + const failAddress = addressOf("fail"); + const okAddress = addressOf("ok"); + + // Drive both outcome classes through the full production path. The + // failing call still completes for the caller — that is exactly why + // the exported span is the only place an operator can see it. + for (const address of [okAddress, failAddress]) { + const execution = yield* client.executions.execute({ + payload: { code: `return await ${address}({});` }, + }); + expect(execution.status, `the ${address} execution completes`).toBe("completed"); + } + + // The failure: outcome attributes on the tool span... + const failSpan = yield* telemetry.expectSpan({ + operation: "executor.tool.execute", + attributes: { "mcp.tool.name": failAddress }, + }); + expect(failSpan.span.tags, "a failed tool call is marked on the exported span").toMatchObject( + { + "executor.tool.outcome": "fail", + "executor.tool.error_code": "upstream_http_error", + "executor.tool.error_status": "502", + }, + ); + expect( + failSpan.span.tags["executor.tenant"], + "the span carries tenant attribution (no trace-join needed to ask 'whose error?')", + ).toBeTruthy(); + + // ...and the upstream status on the HTTP span queries actually target. + const invokeSpan = yield* telemetry.expectSpan({ + operation: "plugin.openapi.invoke", + attributes: { "plugin.openapi.base_url": upstream.baseUrl, "http.status_code": "502" }, + }); + expect( + invokeSpan.span.tags["plugin.openapi.method"], + "the invoke span names the method", + ).toBe("GET"); + + // The success is distinguishable from the failure. + const okSpan = yield* telemetry.expectSpan({ + operation: "executor.tool.execute", + attributes: { "mcp.tool.name": okAddress }, + }); + expect(okSpan.span.tags["executor.tool.outcome"], "a successful call is marked ok").toBe( + "ok", + ); + }), + ), +); diff --git a/e2e/setup/cloud.globalsetup.ts b/e2e/setup/cloud.globalsetup.ts index 59c4201cd..a49ba17e9 100644 --- a/e2e/setup/cloud.globalsetup.ts +++ b/e2e/setup/cloud.globalsetup.ts @@ -28,6 +28,10 @@ export default async function setup(): Promise<(() => Promise) | void> { // Suite-owned trace store — every run captures distributed traces. const motel = await bootMotel(); + // Publish to the test workers (they inherit this process's env): scenarios + // that assert on exported spans yield the Telemetry service, which exists + // only when this is set. No motel → those scenarios skip, never fail. + if (motel) process.env.E2E_MOTEL_URL = motel.url; const publicUrl = `http://127.0.0.1:${ports.E2E_CLOUD_PORT!}`; let booted; diff --git a/e2e/src/scenario.ts b/e2e/src/scenario.ts index 4e018ab8e..cb233c237 100644 --- a/e2e/src/scenario.ts +++ b/e2e/src/scenario.ts @@ -24,6 +24,7 @@ import { makeApiSurface } from "./surfaces/api"; import { makeBrowserSurface } from "./surfaces/browser"; import { makeCliSurface } from "./surfaces/cli"; import { makeMcpSurface } from "./surfaces/mcp"; +import { makeTelemetrySurface } from "./surfaces/telemetry"; import { completeOAuthConsent, hasOpenCode, makeOpenCodeHome, warmUp } from "./clients/opencode"; import { Api, @@ -35,6 +36,7 @@ import { Restart, RunDir, Target, + Telemetry, TtlControl, } from "./services"; import { buildManifest } from "./viewer/manifest"; @@ -62,7 +64,8 @@ type AllServices = | Billing | OpenCode | TtlControl - | Restart; + | Restart + | Telemetry; /** * What this target on this host can provide. Services beyond the base are @@ -94,6 +97,9 @@ const contextFor = (target: TargetShape, dir: string): Context.Context()("e2e/mcp-oauth") {} /** Marker: billing limits are enforced on this target. */ export class Billing extends Context.Service()("e2e/billing") {} +/** Query the suite's OTLP trace store for spans the target actually exported + * (present when the suite booted motel — E2E_MOTEL_URL). */ +export class Telemetry extends Context.Service()("e2e/telemetry") {} + /** The real OpenCode binary, hermetically driveable (present when installed on this host). */ export interface OpenCodeClient { readonly makeHome: typeof makeOpenCodeHome; diff --git a/e2e/src/surfaces/telemetry.ts b/e2e/src/surfaces/telemetry.ts new file mode 100644 index 000000000..eb3549a12 --- /dev/null +++ b/e2e/src/surfaces/telemetry.ts @@ -0,0 +1,75 @@ +// Telemetry surface: query the suite's motel OTLP store for the spans the +// target ACTUALLY exported. This is how a scenario asserts the observability +// contract end-to-end — not "did the code create a span object" but "did the +// span leave the server, with the attributes production queries depend on". +// The dangerous regression mode here is silence (an attribute stamped on the +// wrong span, an error riding the success channel): absent data looks exactly +// like health, so the contract has to be pinned where the data is read. +import { Effect, Schedule } from "effect"; + +/** One exported span, as motel's /api/spans/search returns it. `tags` is the + * span's attributes with every value stringified. */ +export interface ExportedSpan { + readonly traceId: string; + readonly rootOperationName: string; + readonly span: { + readonly spanId: string; + readonly operationName: string; + readonly status: "ok" | "error"; + readonly durationMs: number; + readonly tags: Readonly>; + }; +} + +export interface SpanQuery { + /** Exact-match attribute filters (motel `attr.=`). */ + readonly attributes?: Readonly>; + readonly operation?: string; + readonly traceId?: string; +} + +export interface TelemetrySurface { + /** One-shot search against the trace store. */ + readonly searchSpans: (query: SpanQuery) => Effect.Effect; + /** Search until at least one span matches. Exporters batch (the app + * flushes ~1s after the request), so arrival is eventually-consistent — + * polling IS the contract: "the span reaches the store, soon". */ + readonly expectSpan: (query: SpanQuery) => Effect.Effect; +} + +export const makeTelemetrySurface = (motelUrl: string): TelemetrySurface => { + const searchSpans = (query: SpanQuery) => + Effect.gen(function* () { + const params = new URLSearchParams({ lookback: "15m", limit: "100" }); + if (query.operation) params.set("operation", query.operation); + if (query.traceId) params.set("traceId", query.traceId); + for (const [key, value] of Object.entries(query.attributes ?? {})) { + params.set(`attr.${key}`, value); + } + const response = yield* Effect.promise(() => fetch(`${motelUrl}/api/spans/search?${params}`)); + if (!response.ok) { + return yield* Effect.fail( + `motel span search responded ${response.status}: ${yield* Effect.promise(() => response.text())}`, + ); + } + const body = (yield* Effect.promise(() => response.json())) as { + readonly data?: readonly ExportedSpan[]; + }; + return body.data ?? []; + }); + + return { + searchSpans, + expectSpan: (query) => + searchSpans(query).pipe( + Effect.filterOrFail( + (spans) => spans.length > 0, + () => `no exported span matched ${JSON.stringify(query)}`, + ), + Effect.map((spans) => spans[0]!), + // ~20s ceiling (40 × 500ms): BatchSpanProcessor flushes ~1s after the + // request and the dev stack drains on waitUntil; slower is a real bug. + Effect.retry(Schedule.both(Schedule.spaced("500 millis"), Schedule.recurs(40))), + ), + }; +}; diff --git a/packages/core/execution/src/telemetry-contract.test.ts b/packages/core/execution/src/telemetry-contract.test.ts deleted file mode 100644 index 60d0b6bb1..000000000 --- a/packages/core/execution/src/telemetry-contract.test.ts +++ /dev/null @@ -1,220 +0,0 @@ -// --------------------------------------------------------------------------- -// Telemetry contracts for the tool-dispatch path. -// -// The dangerous observability failure mode is the signal silently going dark: -// expected tool failures (`ToolResult.fail`) resolve through the Effect -// success channel, so without explicit outcome annotation the tracer records -// a healthy span for a user hitting an upstream error wall — absence of -// error data is indistinguishable from health. These tests drive real -// dispatches through a recording tracer and assert the spans, attributes, -// and error statuses that production queries (Axiom) depend on. A regression -// here fails CI instead of being discovered during the next incident. -// --------------------------------------------------------------------------- - -import { describe, expect, it } from "@effect/vitest"; -import { Data, Effect, Exit } from "effect"; - -import { - AuthTemplateSlug, - ConnectionName, - ConnectionNotFoundError, - ElicitationDeclinedError, - IntegrationSlug, - NoHandlerError, - PluginNotLoadedError, - ToolAddress, - ToolBlockedError, - ToolName, - ToolNotFoundError, - ToolResult, - createExecutor, - definePlugin, - type CredentialProvider, -} from "@executor-js/sdk"; -import { ProviderItemId, ProviderKey } from "@executor-js/sdk"; -import { - makeTestConfig, - runWithRecordingTracer, - spanEndedWithError, -} from "@executor-js/sdk/testing"; -import { makeExecutorToolInvoker } from "./tool-invoker"; - -const TEMPLATE = AuthTemplateSlug.make("apiKey"); -const CONN = ConnectionName.make("main"); -const INTEG = IntegrationSlug.make("upstream"); - -const memoryProvider = (): CredentialProvider => { - const store = new Map(); - return { - key: ProviderKey.make("telemetry-memory"), - writable: true, - get: (id) => Effect.sync(() => store.get(String(id)) ?? null), - set: (id, value) => Effect.sync(() => void store.set(String(id), value)), - list: () => - Effect.sync(() => - Array.from(store.keys()).map((key) => ({ - id: ProviderItemId.make(key), - name: key, - })), - ), - }; -}; - -const EmptyInputJson = { type: "object", properties: {} } as const; - -class TelemetryTestDefect extends Data.TaggedError("TelemetryTestDefect")<{ - readonly message: string; -}> {} - -// One integration, three tools spanning the outcome classes the telemetry -// contract distinguishes: domain success, expected upstream failure -// (success channel), and an infra defect (failure channel). -const telemetryPlugin = definePlugin(() => ({ - id: "telemetry-test" as const, - credentialProviders: [memoryProvider()], - storage: () => ({}), - resolveTools: () => - Effect.succeed({ - tools: [ - { name: ToolName.make("succeeds"), description: "", inputSchema: EmptyInputJson }, - { name: ToolName.make("failsUpstream"), description: "", inputSchema: EmptyInputJson }, - { name: ToolName.make("defects"), description: "", inputSchema: EmptyInputJson }, - ], - }), - invokeTool: ({ toolRow }) => { - if (toolRow.name === "succeeds") { - return Effect.succeed(ToolResult.ok({ fine: true })); - } - if (toolRow.name === "failsUpstream") { - return Effect.succeed( - ToolResult.fail({ - code: "upstream_http_error", - status: 502, - message: "Bad gateway from upstream", - }), - ); - } - return Effect.fail(new TelemetryTestDefect({ message: "database exploded" })); - }, - extension: (ctx) => ({ - seed: () => - ctx.core.integrations.register({ - slug: INTEG, - description: "telemetry upstream", - config: {}, - }), - }), -}))(); - -const makeHarness = () => - Effect.gen(function* () { - const executor = yield* createExecutor(makeTestConfig({ plugins: [telemetryPlugin] as const })); - yield* (executor as never as Record<"telemetry-test", { seed: () => Effect.Effect }>)[ - "telemetry-test" - ].seed(); - yield* executor.connections.create({ - owner: "org", - name: CONN, - integration: INTEG, - template: TEMPLATE, - value: "token", - }); - return makeExecutorToolInvoker(executor, { invokeOptions: {} }); - }); - -const dispatch = (tool: string) => - Effect.gen(function* () { - const invoker = yield* makeHarness(); - return yield* invoker.invoke({ path: `upstream.org.main.${tool}`, args: {} }); - }); - -describe("telemetry contract: tool dispatch spans", () => { - it.effect("a successful tool stamps outcome=ok on dispatch and execute spans", () => - Effect.gen(function* () { - const { exit, recording } = yield* runWithRecordingTracer(dispatch("succeeds")); - expect(Exit.isSuccess(exit)).toBe(true); - - const dispatchSpan = recording.single("mcp.tool.dispatch"); - expect(dispatchSpan.attributes.get("mcp.tool.name")).toBe("upstream.org.main.succeeds"); - expect(dispatchSpan.attributes.get("mcp.tool.integration")).toBe("upstream"); - expect(dispatchSpan.attributes.get("executor.tool.outcome")).toBe("ok"); - - const executeSpan = recording.single("executor.tool.execute"); - expect(executeSpan.attributes.get("executor.tool.outcome")).toBe("ok"); - // Org/tenant attribution on the tool span itself — production queries - // must not need a trace-id join against the outer request span. - expect(executeSpan.attributes.get("executor.tenant")).toBe("test-tenant"); - expect(executeSpan.attributes.get("executor.subject")).toBe("test-subject"); - }), - ); - - it.effect( - "an expected upstream failure (success channel) stamps outcome=fail + code + status", - () => - Effect.gen(function* () { - const { exit, recording } = yield* runWithRecordingTracer(dispatch("failsUpstream")); - // The contract under test: the failure is a VALUE, not an Effect error… - expect(Exit.isSuccess(exit)).toBe(true); - - // …so the span must carry the outcome explicitly, on both the - // sandbox-dispatch span and the executor-execute span. - for (const name of ["mcp.tool.dispatch", "executor.tool.execute"]) { - const span = recording.single(name); - expect(span.attributes.get("executor.tool.outcome")).toBe("fail"); - expect(span.attributes.get("executor.tool.error_code")).toBe("upstream_http_error"); - expect(span.attributes.get("executor.tool.error_status")).toBe(502); - } - }), - ); - - it.effect("an infra defect ends the dispatch span with an error exit", () => - Effect.gen(function* () { - const { exit, recording } = yield* runWithRecordingTracer(dispatch("defects")); - expect(Exit.isFailure(exit)).toBe(true); - - const dispatchSpan = recording.single("mcp.tool.dispatch"); - expect(spanEndedWithError(dispatchSpan)).toBe(true); - }), - ); - - it.effect("tool_not_found surfaces as outcome=fail with its code on the dispatch span", () => - Effect.gen(function* () { - const { exit, recording } = yield* runWithRecordingTracer(dispatch("doesNotExist")); - // tool_not_found is an expected failure: surfaced as a ToolResult.fail - // VALUE through the success channel. - expect(Exit.isSuccess(exit)).toBe(true); - - const dispatchSpan = recording.single("mcp.tool.dispatch"); - expect(dispatchSpan.attributes.get("executor.tool.outcome")).toBe("fail"); - expect(dispatchSpan.attributes.get("executor.tool.error_code")).toBe("tool_not_found"); - }), - ); -}); - -describe("telemetry contract: error messages", () => { - it("sdk tagged errors render a non-empty message for span status", () => { - // These messages become OTLP status.message via Cause.prettyErrors — - // before the derived getters, 562 of 804 error spans in a week of prod - // data had an EMPTY status message (the error class defined no - // `message`, and TaggedErrorClass instances default to ""). - const address = ToolAddress.make("tools.upstream.org.main.failsUpstream"); - const errors: ReadonlyArray = [ - new ToolNotFoundError({ address }), - new ToolBlockedError({ address, pattern: "*" }), - new PluginNotLoadedError({ address, pluginId: "p" }), - new NoHandlerError({ address, pluginId: "p" }), - new ConnectionNotFoundError({ - owner: "org", - integration: INTEG, - name: CONN, - }), - new ElicitationDeclinedError({ address, action: "decline" }), - ]; - for (const error of errors) { - // oxlint-disable-next-line executor/no-unknown-error-message -- the test asserts ON the message contract itself - const { message, name } = error; - expect(message.length, `${name} must derive a message`).toBeGreaterThan(0); - expect(message).not.toContain("[object Object]"); - } - }); -}); diff --git a/packages/core/sdk/src/testing.ts b/packages/core/sdk/src/testing.ts index 9e51c2f6d..483a0fcfd 100644 --- a/packages/core/sdk/src/testing.ts +++ b/packages/core/sdk/src/testing.ts @@ -37,14 +37,6 @@ export { type OutputTypeScriptContract, type TypeCheckOutputTypeScriptOptions, } from "./testing/tool-output-contract"; -export { - makeRecordingTracer, - runWithRecordingTracer, - spanEndedWithError, - spanExit, - type RecordedSpan, - type RecordingTracer, -} from "./testing/recording-tracer"; export class TestHttpServerAddressError extends Data.TaggedError("TestHttpServerAddressError")<{ readonly address: unknown; diff --git a/packages/core/sdk/src/testing/recording-tracer.ts b/packages/core/sdk/src/testing/recording-tracer.ts deleted file mode 100644 index 1a7d2a72a..000000000 --- a/packages/core/sdk/src/testing/recording-tracer.ts +++ /dev/null @@ -1,80 +0,0 @@ -import { Effect, Exit, Predicate, Tracer } from "effect"; - -/* Telemetry contract testing: a Tracer that records every span it creates so - * tests can assert spans, attributes, and error statuses actually exist for a - * given operation. The product's biggest observability failure mode is the - * signal silently going dark (a span attribute set after the wrong span, an - * error riding the success channel) — absence of data looks identical to - * health in production, so these contracts have to be pinned in tests. */ - -export interface RecordedSpan { - readonly name: string; - readonly span: Tracer.NativeSpan; - /** Attributes as a plain object snapshot (live map — read after run). */ - readonly attributes: ReadonlyMap; -} - -export interface RecordingTracer { - readonly tracer: Tracer.Tracer; - readonly spans: readonly RecordedSpan[]; - /** All recorded spans with the given name. */ - readonly byName: (name: string) => readonly RecordedSpan[]; - /** Exactly-one convenience: throws when zero or multiple spans match. */ - readonly single: (name: string) => RecordedSpan; -} - -export const makeRecordingTracer = (): RecordingTracer => { - const spans: RecordedSpan[] = []; - const tracer = Tracer.make({ - span: (options) => { - const span = new Tracer.NativeSpan(options); - spans.push({ name: options.name, span, attributes: span.attributes }); - return span; - }, - }); - return { - tracer, - spans, - byName: (name) => spans.filter((entry) => entry.name === name), - single: (name) => { - const matches = spans.filter((entry) => entry.name === name); - if (matches.length !== 1) { - // oxlint-disable-next-line executor/no-try-catch-or-throw, executor/no-error-constructor -- test helper: a wrong span count is a test bug, fail the test loud - throw new Error( - `Expected exactly one "${name}" span, recorded ${matches.length} (all spans: ${spans.map((entry) => entry.name).join(", ")})`, - ); - } - return matches[0]!; - }, - }; -}; - -/** Run an effect with a recording tracer and return both its exit and the - * recorded spans. The effect's failure is captured, not thrown — telemetry - * contracts usually assert on spans of FAILING operations. */ -export const runWithRecordingTracer = ( - effect: Effect.Effect, -): Effect.Effect<{ - readonly exit: Exit.Exit; - readonly recording: RecordingTracer; -}> => - Effect.suspend(() => { - const recording = makeRecordingTracer(); - return Effect.exit(effect).pipe( - Effect.withTracer(recording.tracer), - Effect.map((exit) => ({ exit, recording })), - ); - }); - -/** The span's ended exit, or null while started. */ -export const spanExit = (entry: RecordedSpan): Exit.Exit | null => { - const status = entry.span.status; - return Predicate.isTagged(status, "Ended") ? status.exit : null; -}; - -/** True when the span ended with a failure exit (what OTLP exporters map to - * status ERROR). */ -export const spanEndedWithError = (entry: RecordedSpan): boolean => { - const exit = spanExit(entry); - return exit != null && Exit.isFailure(exit); -}; diff --git a/packages/plugins/openapi/src/sdk/upstream-failures.test.ts b/packages/plugins/openapi/src/sdk/upstream-failures.test.ts index 42e2be694..ba2389423 100644 --- a/packages/plugins/openapi/src/sdk/upstream-failures.test.ts +++ b/packages/plugins/openapi/src/sdk/upstream-failures.test.ts @@ -32,11 +32,7 @@ import { IntegrationSlug, ToolAddress, } from "@executor-js/sdk"; -import { - makeTestConfig, - memoryCredentialsPlugin, - runWithRecordingTracer, -} from "@executor-js/sdk/testing"; +import { makeTestConfig, memoryCredentialsPlugin } from "@executor-js/sdk/testing"; import { addOpenApiTestConnection, makeOpenApiHttpApiTestSourceConfig, @@ -272,56 +268,6 @@ describe("OpenAPI upstream failure modes", () => { }), ); - // Telemetry contract: the spans production queries depend on must carry - // the upstream outcome. This pins two real prod regressions: (1) - // `http.status_code` was annotated inside `OpenApi.invoke` but queries - // target the `plugin.openapi.invoke` wrapper span, which carried it on - // 0 of 19.5k spans; (2) ToolResult.fail rides the success channel, so - // without outcome attributes a 5xx-spewing integration looks healthy. - it.effect("a failing invocation stamps status + outcome on its spans", () => - Effect.gen(function* () { - const server = yield* startScriptedServer(() => ({ - status: 502, - headers: { "content-type": "application/json" }, - body: '{"error":{"message":"bad gateway"}}', - })); - const { executor, address } = yield* buildExecutorForOpenApiServer(server); - - const { exit, recording } = yield* runWithRecordingTracer(executor.execute(address, {})); - expect(Exit.isSuccess(exit)).toBe(true); - - const invokeSpan = recording.single("plugin.openapi.invoke"); - expect(invokeSpan.attributes.get("http.status_code")).toBe(502); - expect(invokeSpan.attributes.get("plugin.openapi.method")).toBe("GET"); - - const executeSpan = recording.single("executor.tool.execute"); - expect(executeSpan.attributes.get("executor.tool.outcome")).toBe("fail"); - expect(executeSpan.attributes.get("executor.tool.error_code")).toBe("upstream_http_error"); - expect(executeSpan.attributes.get("executor.tool.error_status")).toBe(502); - expect(executeSpan.attributes.get("executor.tenant")).toBe("test-tenant"); - }), - ); - - it.effect("a successful invocation stamps status + outcome=ok on its spans", () => - Effect.gen(function* () { - const server = yield* startScriptedServer(() => ({ - status: 200, - headers: { "content-type": "application/json" }, - body: "[]", - })); - const { executor, address } = yield* buildExecutorForOpenApiServer(server); - - const { exit, recording } = yield* runWithRecordingTracer(executor.execute(address, {})); - expect(Exit.isSuccess(exit)).toBe(true); - - const invokeSpan = recording.single("plugin.openapi.invoke"); - expect(invokeSpan.attributes.get("http.status_code")).toBe(200); - - const executeSpan = recording.single("executor.tool.execute"); - expect(executeSpan.attributes.get("executor.tool.outcome")).toBe("ok"); - }), - ); - it.effect("upstream slow-then-respond doesn't lose the request", () => Effect.gen(function* () { const slowServer = Effect.acquireRelease( From e9dda2886a5b9edf5f0416ba8c1cc95a4bb8191c Mon Sep 17 00:00:00 2001 From: Rhys Sullivan Date: Fri, 12 Jun 2026 16:28:32 -0700 Subject: [PATCH 4/4] Document the telemetry surfaces for future work MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - prod-telemetry repo skill: the executor-cloud Axiom dataset layout (attributes.custom JSON map, span names and their attributes including the new outcome fields), working APL recipes for error digests and org attribution, prod-DB and PostHog access notes, and the deploy-canary procedure. - e2e/AGENTS.md: the Telemetry service — scenarios can assert on the spans the target actually exported, with the arrival-polling and fixture-tagging gotchas. --- .claude/skills/prod-telemetry/SKILL.md | 114 +++++++++++++++++++++++++ e2e/AGENTS.md | 25 ++++++ 2 files changed, 139 insertions(+) create mode 100644 .claude/skills/prod-telemetry/SKILL.md diff --git a/.claude/skills/prod-telemetry/SKILL.md b/.claude/skills/prod-telemetry/SKILL.md new file mode 100644 index 000000000..47f79bbc7 --- /dev/null +++ b/.claude/skills/prod-telemetry/SKILL.md @@ -0,0 +1,114 @@ +--- +name: prod-telemetry +description: Query Executor's production telemetry — Axiom traces (executor-cloud dataset), prod Postgres via PlanetScale, PostHog product analytics — through the Executor MCP. Use when investigating prod errors, latency, usage, churn signals, or verifying a deploy's telemetry; includes the dataset field layout, working APL recipes, and the error-attribution join. +--- + +# Production telemetry access + +All three stores are queryable through the Executor MCP's connected +integrations — no dashboards or credentials needed. Verify the connection +exists with `connections.list` if a call fails. + +## Axiom traces (`axiom_mcp`) + +Tool: `axiom_mcp.user.axiomMcpOAuth.querydataset` — the argument is `apl` +(NOT `query`). Dataset: `['executor-cloud']` (worker spans; browser spans +join the same traces via traceparent). + +**Field layout (the part you'd otherwise rediscover by failed queries):** + +- Custom span attributes live under the JSON map `['attributes.custom']`, + NOT as top-level `attributes.*` columns. Read with + `['attributes.custom']['mcp.tool.name']`. A nonexistent top-level field is + a hard query error ("invalid field"), not an empty result. +- Span status: `['status.code']` (`"OK"`/`"ERROR"`), `['status.message']`. +- Exceptions: the `events` column carries `exception.type` / + `exception.stacktrace` JSON. +- OTel basics are top-level: `name`, `trace_id`, `span_id`, + `parent_span_id`, `duration`, `_time`. + +**Span names worth querying** (and their custom attrs): + +- `executor.tool.execute` — `mcp.tool.name` (full address), and since + PR #992: `executor.tool.outcome` (`ok`/`fail`), + `executor.tool.error_code`, `executor.tool.error_status`, + `executor.tenant`, `executor.subject`. +- `mcp.tool.dispatch` — `mcp.tool.name` (sandbox path), + `mcp.tool.integration`, same outcome attrs. +- `plugin.openapi.invoke` — `plugin.openapi.method` / `path_template` / + `base_url`, and since PR #992 `http.status_code`. +- `mcp.request` (outer) — `mcp.auth.organization_id`, + `mcp.auth.account_id`, `mcp.tool.name`, CF edge fields (`cf.country`…), + MCP client fingerprint (`mcp.client.name`…). + +**Recipe — error signatures by class (the daily-digest query):** + +```apl +['executor-cloud'] +| where _time > ago(1d) +| where ['status.code'] == "ERROR" and name == "executor.tool.execute" +| extend msg = substring(tostring(['status.message']), 0, 120) +| extend tool = tostring(['attributes.custom']['mcp.tool.name']) +| summarize n = count() by msg, tool +| sort by n desc +``` + +**Recipe — attribute errors to orgs.** Tool spans now carry +`executor.tenant` directly (post-#992). For spans from BEFORE that deploy, +join through the outer request span: + +```apl +['executor-cloud'] +| where name == "mcp.request" and isnotnull(['attributes.custom']['mcp.auth.organization_id']) +| project trace_id, org = tostring(['attributes.custom']['mcp.auth.organization_id']) +| join kind=inner ( + ['executor-cloud'] + | where ['status.code'] == "ERROR" and name == "executor.tool.execute" + | project trace_id, msg = substring(tostring(['status.message']), 0, 60) + ) on trace_id +| summarize n = count() by org, msg | sort by n desc +``` + +**Recipe — upstream failure rate per integration (post-#992 attrs):** + +```apl +['executor-cloud'] +| where _time > ago(1d) and name == "mcp.tool.dispatch" +| extend outcome = tostring(['attributes.custom']['executor.tool.outcome']) +| extend integration = tostring(['attributes.custom']['mcp.tool.integration']) +| where isnotnull(outcome) +| summarize calls = count(), fails = countif(outcome == "fail") by integration +| extend failRate = todouble(fails) / todouble(calls) +| sort by fails desc +``` + +**Known signal caveats** (audited 2026-06-12): + +- Pre-#992 spans: `ToolResult.fail` outcomes (upstream 4xx/5xx, auth + rejections) are INVISIBLE — they rode the Effect success channel with no + span marker. Don't conclude "no errors" from old data. +- Many pre-#992 ERROR spans have an EMPTY `status.message` (tagged errors + without a message field) — group those by `events` exception.type instead. +- `[object Object]` status messages are the pre-#992 formatting bug. + +## Prod database (`planetscale_mcp`) + +Read tool needs `{organization: "answer-overflow", database: "executor", +branch: "main"}`. It returns `ok: true` even when the SQL failed — check the +result text for `Error:`. Use for tenant/integration/connection facts that +spans don't carry (row sizes, config shapes, counts). + +## Product analytics (`posthog_api` / `mcp_posthog_com`) + +Browser-side events only (the ~60-event typed catalog, PR #987; server-side +events not built). The org-key `posthog_api` connection covers the REST API; +the OAuth MCP connection covers the higher-level tools. + +## Verifying a deploy's telemetry (Layer-0 canary) + +After deploying telemetry changes: run a known-failing tool call against +prod, then assert the expected attributes arrive in Axiom within ~1 min. +Absence of data looks identical to health — query for the NEW attribute +explicitly rather than eyeballing dashboards. The e2e equivalent runs on +every suite: `e2e/cloud/telemetry-contract.test.ts` via the `Telemetry` +service (motel `/api/spans/search?attr.=`). diff --git a/e2e/AGENTS.md b/e2e/AGENTS.md index 9a97bed3c..a77694871 100644 --- a/e2e/AGENTS.md +++ b/e2e/AGENTS.md @@ -77,6 +77,31 @@ const r = yield * session.call("execute", { code: "return 1 + 1;" }); // human-in-the-loop: session.approvePaused(r.text) resumes a paused execution ``` +## Telemetry scenarios (cloud) + +The suite boots a motel OTLP store and points the target's real exporter at +it, so a scenario can assert on the spans the server ACTUALLY exported — +the layer where "observability silently went dark" bugs live (an attribute +stamped on a span the exporter never carries looks identical to health). + +```ts +const telemetry = yield * Telemetry; // skips when motel didn't boot +const span = + yield * + telemetry.expectSpan({ + operation: "executor.tool.execute", + attributes: { "mcp.tool.name": failAddress }, // exact match, values stringified + }); +expect(span.span.tags["executor.tool.outcome"]).toBe("fail"); +``` + +- `expectSpan` polls (~20s): exporters batch, so arrival is + eventually-consistent — "the span reaches the store, soon" IS the contract. +- Spec gotcha for fixtures: give operations explicit `tags` — tool addresses + are `group.leaf`, and an untagged op derives its group from the URL path, + so `/fail` does NOT produce a `.fail`-suffixed address. +- Prior art: `cloud/telemetry-contract.test.ts`. + ## Running ```sh