Browser/LinkedIn safety: queue, rate limits, emergency stop, cancellation, health autostop - #1
Open
DominikLed wants to merge 34 commits into
Open
Browser/LinkedIn safety: queue, rate limits, emergency stop, cancellation, health autostop#1DominikLed wants to merge 34 commits into
DominikLed wants to merge 34 commits into
Conversation
…iles Install supabase and supabase-postgres-best-practices skills via `npx skills add supabase/agent-skills` (.agents/ + skills-lock.json). Ignore the root node_modules/, package.json, package-lock.json pulled in by the installer, plus Next.js .next/ build output. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…et, retry policy, manual control Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
CampaignJob and EnrichmentJob also carry JobStatus-based CheckConstraints, so the shared-enum change needs their migrations too (keeps makemigrations --check clean across all apps). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Parity tests for the shared-enum constraint migrations added in e9e2155, closing the Task 1 review's Important finding. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ts, retry, autostop) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Baseline commit of the previously-uncommitted onboarding/auth work that the browser-safety feature builds on: UserProfile onboarding state (PENDING/READY/ DISABLED), Workspace/SenderAccount/SearchCriteria assignment, IsReady gate on browser + campaign triggers, JWT email-verified claim resolution, and their tests + migration 0005. Scoped to backend Python only; frontend onboarding UI, the webbridge-e2e marker, and the DB CONN_HEALTH_CHECKS tweak remain as separate uncommitted WIP. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ation; drop unused import
…alth, retry, timeout)
…or body A daemon HTTP 502 (e.g. the automation tab was closed mid-job: "session tab was closed — navigate first to recreate") failed a BrowserJob outright with an opaque "daemon returned HTTP 502 for evaluate": _command discarded the daemon's JSON error body and raised an untyped KimiWebBridgeError, which bypassed the RetryableBrowserError path in browser/tasks.py and landed in the generic hard-fail branch. - Add KimiWebBridgeTransportError(KimiWebBridgeError, RetryableBrowserError) for transient faults (daemon unreachable, timeout, 5xx) so tasks.py re-queues with backoff per the taxonomy in services/browser/exceptions.py. - Classify connect/timeout errors and daemon 5xx as transport (retryable); keep 4xx a fatal KimiWebBridgeError so a request-level bug can't loop. - _daemon_error_detail surfaces the daemon's real message so a 502 is diagnosable instead of an opaque status code. Verified against the live daemon and with 3 new tests; existing daemon-error tests still pass (the new type is-a KimiWebBridgeError). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Audit outcome: the Kimi provider is a daemon+extension client with no Playwright/Chromium launch code, so there were no temp-profile / incognito / new_context / cookie-clearing bugs to fix — the persistent profile is already reused for every job. The real gap was diagnostics, plus hardening the one place Chrome is launched. KimiBrowserProvider._assert_linkedin_session now emits a structured preflight log — profile path, linkedin.com cookie count, li_at presence, landing URL, authenticated verdict — and logs that the persistent profile is REUSED (never recreated). New best-effort _linkedin_cookie_state reads linkedin.com cookies via CDP Network.getCookies; it never gates or raises (the navigation-based auth-wall check stays the source of truth). The logged-out path keeps failing fast with the same actionable "log in manually" message; the provider still never enters credentials or 2FA. launch_kimi_chrome.ps1: refuse an empty or temp-rooted --user-data-dir (would silently drop the LinkedIn session), log REUSING-vs-CREATED profile and whether a Chrome Cookies file is present, and reassert no incognito/guest. Verified live: preflight read 24 cookies, li_at present, authenticated. Adds provider tests for the cookie helper and preflight logging; full browser suite (95 tests) green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A runaway criteria set (the live failure carried ~40 phrases, including duplicates like "logistics logistics" from overlapping job_position and industry inputs) built a multi-kilobyte Sales Navigator URL that the tab could not render, and the daemon reported it "closed" mid-job. _build_search_url now drops repeated phrases case-insensitively (casefold, correct for non-ASCII EU job titles) and caps the result at KIMI_WEBBRIDGE_MAX_KEYWORDS (default 25, env-overridable) so the URL stays short. Covered by two new provider URL tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
useBrowserJobStatusLive derived its terminal check from the initial Supabase row status, which is fetched once and never refetched while the page is open. Once a job finished, the live endpoint reported succeeded/failed but the stale row kept refetchInterval at 2s, so the page polled forever. Extract browserJobPollInterval(live, fallback), which keys the stop condition on the freshly-fetched live status (falling back to the row only until the first live response arrives), and drive refetchInterval from query.state.data. Pure function, unit-tested in hooks.test.tsx. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
dashboard/page.tsx and search/page.tsx already import @/components/onboarding-banner, but the component file was never committed, so a fresh checkout of this branch failed to build. Add the component (renders a PENDING-user onboarding banner keyed off profile status). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
conftest.py clears Django LocMemCache before and after each test. The browser-safety primitives (single-session lock, emergency stop, per-user action budget/cooldown) keep hot state in the cache, which is not rolled back between tests. A leaked lock made run_browser_job re-queue itself synchronously under CELERY_TASK_ALWAYS_EAGER until RecursionError poisoned the DB connection - passing in isolation, failing in the full suite. Clearing around every test removes the order dependency. tests/test_auth_e2e_webbridge.py drives the real login -> /dashboard auth flow through Kimi WebBridge and asserts a healthy page (no console errors, no failed requests, /api/activity 200, onboarding banner shown). It auto-skips unless E2E_WEBBRIDGE=1 and the daemon/servers are reachable, so normal runs stay green; it creates and tears down a throwaway Supabase user per run. Documented in docs/testing/webbridge-e2e.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Trigger view (Task 13): before creating a job, refuse while the global emergency stop is on (409) and enforce the per-user action budget / cooldown (429, with Retry-After on cooldown). The worker re-checks both; this is the fast fail so a rate-limited user never enqueues. Rate-limit refusals are logged (browser_job.rate_limited). Cancel endpoint (Task 14): POST /api/browser-jobs/<pk>/cancel/. PENDING → CANCELLED immediately; RUNNING → the cooperative cancel flag the worker honors between pages; already-terminal → 409; not-owner/missing → 404. Ownership scoped by requested_by, re-applying RLS at the app layer. TDD: browser/test_views_safety.py (gate + cancel). The allowed-trigger test mocks enqueue so it exercises the gate, not the eager pipeline. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
KimiWebBridgeSessionError now also subclasses SessionHealthError, so an auth-wall/checkpoint landing routes through browser/tasks._handle_health (non-retryable; a checkpoint/captcha kind escalates to the global emergency stop) instead of the generic failure path, which never triggered autostop. The kind is the classify_page verdict of the landed URL (checkpoint vs logged_out), defaulting to logged_out for auth-wall markers the classifier doesn't name (e.g. /authwall). Both raise sites (preflight + mid-search auth-wall guard) go through one helper. Transport/timeout faults already mapped to RetryableBrowserError via KimiWebBridgeTransportError, so Task 15's retry half was already done. TDD: KimiSessionHealthErrorTests drives the real _FakeDaemon preflight (not synthetic mocks) for logged_out, checkpoint, and /authwall kinds. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…shboard Register BrowserControlState (the emergency-stop toggle) in admin. Enhance BrowserJobAdmin with a duration column, a "Cancel selected" action (PENDING/RUNNING → CANCELLED, setting the cancel flag for running jobs), and a staff-only "LinkedIn Queue" operational dashboard at /admin/browser/browserjob/linkedin-queue/ showing active/pending jobs, per-user daily usage vs budget, emergency-stop state, and session-lock holder. TDD: browser/test_admin.py (staff gate, job listing, cancel action). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ancel Document the operator controls shipped in Tasks 13-16: single-session worker concurrency, the emergency-stop toggle, per-user action budgets and the 429 trigger refusal, the cancel endpoint, the LinkedIn Queue admin dashboard, and health auto-stop on captcha/checkpoint. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
request_with_retry replays the identical request, so only transient failures are worth another attempt: transport errors (timeout, connection refused, DNS), HTTP 429 (Apollo's rate-limit signal), and any 5xx. A deterministic non-2xx — 4xx bad key/request, or an unfollowed 3xx — fails identically on retry and only burns quota, so it gives up immediately. Replaces the never-retry placeholder contribution point. TDD: services/common/test_http.py covers each branch. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Runs on push to main and every PR: installs deps against a Postgres 16 service, checks for missing migrations, runs the pytest suite, and runs Django's deployment security checks under prod settings. The suite is hermetic in CI: no APOLLO/RICHAPI keys are set, and both enrichment clients short-circuit without a key (no live HTTP). To let Django reach the SSL-less service container, DATABASES sslmode is now overridable via SUPABASE_DB_SSLMODE (defaults to 'require' — prod and local Supabase connections are unchanged). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The trigger view now enforces a per-user action cooldown (429). The discovery-pipeline suite triggers back-to-back searches to verify rediscovery dedup — not rate limiting — so its fixture profile now sets action_cooldown_seconds=0. The gate itself is covered by browser/test_views_safety. Regression surfaced by the full suite after the Task 13 trigger gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the full browser/LinkedIn-safety plan
(docs/superpowers/plans/2026-07-09-browser-linkedin-safety.md) plus the
onboarding baseline, a live frontend, CI, and supporting test infra.
30 commits, 74 files, +5006/-199. Every unit is TDD; full backend suite
is green (310 passed, 1 opt-in skip).
Safety controls (spec §A–J)
worker refuses/cancels, trigger endpoint returns 409, admin toggle.
POST /api/browser-jobs/<pk>/cancel/(PENDING→CANCELLED, RUNNING→flag).trigger endpoint 429s over-budget (Retry-After on cooldown).
shared
SessionHealthError(kind), and captcha/checkpoint auto-engage theemergency stop. Transport/timeout faults map to
RetryableBrowserError.should_retryretries transport errors / 429 / 5xx only.Also in this branch
the tab); daemon 5xx retry; session-persistence diagnostics.
Verification
pytest -q→ 310 passed, 1 skipped (opt-in e2e).makemigrations --check→ no changes.check --deploy→ clean (W009 only).Notes for reviewers
PaginatableTS error infrontend/src/lib/supabase/queries.test.ts(not touched here).🤖 Generated with Claude Code