Skip to content

fix(cli): stop error-tracking expected user errors and per-version fingerprints - #2885

Draft
posthog-eu[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixcli-stop-reporting-expected-user-1ec900
Draft

fix(cli): stop error-tracking expected user errors and per-version fingerprints#2885
posthog-eu[bot] wants to merge 1 commit into
mainfrom
posthog-self-driving/fixcli-stop-reporting-expected-user-1ec900

Conversation

@posthog-eu

@posthog-eu posthog-eu Bot commented Aug 6, 2026

Copy link
Copy Markdown

Summary

Ordinary user mistakes from the CLI (bad/missing API key, app not created yet) were being reported to error tracking and re-fingerprinted on every release, so the same errors kept spawning brand-new issues instead of one triageable issue.

Why: real signal was shredded into noise — the bundle upload org-lookup failure alone fired ~1.4K times over 90 days, split across 40+ per-version fingerprints, and every CLI release triggered a fresh round of "new issue" alerts for problems that have existed for months. Users hit the same path got an opaque message with no hint about what to do.

Two root causes, both in the CLI:

  • Version in the fingerprintcapturePosthogException folded cli:<version>:<command> into $exception_fingerprint, so every release re-fingerprinted identical errors into a fresh issue.
  • Everything captured — the top-level handler sent all non-Commander throws to error tracking, including plain user-configuration failures.

Plus, getOrganizationId discarded its formatError(error) result (losing the real PostgREST cause), and bundle upload ran the org lookup before the app-existence/permission check — so a missing app failed with Cannot get organization id instead of the actionable "run app add" message.

Changes

Area Before After
posthog.ts fingerprint includes CLI version → new issue per release version-less (cli:<command>:…); version still sent as cli_version
index.ts top-level handler captures all non-Commander errors skips capture for expected user errors (401 invalid_apikey/no_key_provided, app-not-found); still calls trackCommandFailed
getOrganizationId formatError(error) result discarded underlying cause logged alongside the message
bundle upload order org lookup → (much later) app/permission check app existence + upload permission checked before org lookup

Expected user errors are identified by a new isExpectedUserError helper (message markers + 401 status, including supabase-js FunctionsHttpError.context.status). Genuinely unexpected failures — including a real getOrganizationId throw for an app that does exist — are still captured.

Test plan

  • bun test/test-posthog-exception.mjs — extended to assert the fingerprint no longer contains the version and that isExpectedUserError classifies the invalid-key / no-key / app-not-found / 401 cases as expected and other errors as not.
  • bun run lint and bun run typecheck pass.

Screenshots

N/A — telemetry/error-classification change, no visible CLI output change beyond the org-lookup error now including its cause.

Checklist

  • My code follows the code style of this project and passes bun run lint.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • My change has adequate E2E test coverage.
  • I have tested my code manually, and I have provided steps how to reproduce my tests.

Created with PostHog Desktop from this inbox report.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Review in cubic

…ngerprints

- Drop the CLI version from the exception fingerprint so one error is one issue
  across releases (version still reported via cli_version).
- Skip exception capture for expected user errors (401 invalid_apikey /
  no_key_provided, app-not-found) while still counting them via trackCommandFailed.
- Log the underlying PostgREST cause in getOrganizationId instead of discarding it.
- Verify app existence + upload permission before the org lookup in bundle upload
  so users get an actionable message instead of the opaque org-id throw.

Generated-By: PostHog Code
Task-Id: cca2ec80-cecd-4b4a-8f45-e50058b4206f
@codspeed-hq

codspeed-hq Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 43 untouched benchmarks
⏩ 2 skipped benchmarks1


Comparing posthog-self-driving/fixcli-stop-reporting-expected-user-1ec900 (f28e31d) with main (60a641e)

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

@sonarqubecloud

sonarqubecloud Bot commented Aug 6, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants