Skip to content

fix(frontend): auto-retry transient Edge Function failures - #2891

Draft
posthog-eu[bot] wants to merge 2 commits into
mainfrom
posthog-self-driving/fixfrontend-investigate-recurring-edge-21d12b
Draft

fix(frontend): auto-retry transient Edge Function failures#2891
posthog-eu[bot] wants to merge 2 commits into
mainfrom
posthog-self-driving/fixfrontend-investigate-recurring-edge-21d12b

Conversation

@posthog-eu

@posthog-eu posthog-eu Bot commented Aug 6, 2026

Copy link
Copy Markdown

Summary

  • Console traffic shares one invocation path — invokeCapgoApi (88 call sites). When the backend is under stress (DB connection-pool exhaustion → DrizzleQueryError / pool timeouts), that path failed immediately with no recovery, surfacing as console errors across dashboard load, usage, org access, API keys, webhooks and device flows.
  • Frontend logs over the last 8 days show the two failure signatures are transient and safe to retry:
Signature Cause Retryable
Failed to send a request to the Edge Function / Failed to fetch network drop — request never completed yes
Edge Function returned a non-2xx status code (5xx/429) backend momentarily overloaded yes (5xx/429 only)
  • The same user was seen manually reloading a failed dashboard fetch repeatedly — there was no automatic retry.
  • Fix: centralized retry with jittered exponential backoff in invokeCapgoApi, covering both the Cloudflare fetch path and the self-host supabase.functions.invoke path.
    • Idempotent requests (GET/HEAD) retry up to 2 times by default; mutations never auto-retry so a retry can't double-apply.
    • Only transient errors retry — 4xx business errors surface immediately, unchanged.
    • Per-call override via a new retries option; default behaviour for mutations is byte-for-byte the same (single attempt).

Why: a shared availability regression was producing dozens of silent, immediately-failing requests per day across core workflows; retrying transient failures lets them recover before the user ever sees an error, which should pull the daily failure-log count down from its current baseline.

Note

This is a frontend resilience fix for the symptom (silent, un-retried failures). The upstream trigger — database connection-pool exhaustion — is a separate backend concern and is not addressed here.

Test plan

  • bun test:unit — added tests/capgo-api-retry.unit.test.ts covering the retry policy (status classification, idempotent-only defaults, transient-error detection, backoff bounds). All pass. bun lint and bun typecheck clean.
  • Post-merge: watch the daily "Edge Function failures by day" count fall from the ~53/day baseline.

Screenshots

Checklist

  • My code follows the code style of this project and passes
    bun run lint:backend && bun run lint.
  • My change requires a change to the documentation.
  • I have updated the documentation
    accordingly.
  • My change has adequate E2E test coverage.
  • I have tested my code manually, and I have provided steps how to reproduce
    my tests

Created with PostHog Desktop from this inbox report.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.

Review in cubic

Console traffic shares one invocation path (invokeCapgoApi, 88 call sites).
Transient failures there — network drops ("Failed to send a request to the
Edge Function") and 5xx/429 responses from a stressed backend (DB connection
-pool exhaustion) — surfaced immediately as console errors across dashboard,
usage, access, webhook and device flows, with no recovery path.

Add centralized retry with jittered exponential backoff for transient
failures. Idempotent requests (GET/HEAD) retry up to 2 times by default;
mutations never auto-retry so a retry cannot double-apply. Applies to both
the Cloudflare fetch path and the self-host supabase.functions.invoke path.

Generated-By: PostHog Code
Task-Id: 8cfe4314-5c96-485b-9aed-921b030b50f9
@codspeed-hq

codspeed-hq Bot commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 43 untouched benchmarks
⏩ 2 skipped benchmarks1


Comparing posthog-self-driving/fixfrontend-investigate-recurring-edge-21d12b (d631ca0) with main (60a641e)

Open in CodSpeed

Footnotes

  1. 2 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports.

Comment thread src/services/capgoApi.ts Fixed
SonarCloud flagged Math.random() in retryBackoffMs as a PRNG-in-security
-context hotspot, failing the quality gate. The value is only backoff jitter,
but switch to globalThis.crypto.getRandomValues (the pattern used elsewhere in
the app) to clear the gate while keeping the anti-thundering-herd jitter.

Generated-By: PostHog Code
Task-Id: 8cfe4314-5c96-485b-9aed-921b030b50f9
@sonarqubecloud

sonarqubecloud Bot commented Aug 6, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant