Skip to content

Sentry log enrichment for backend Workers#568

Merged
isuttell merged 3 commits into
mainfrom
codex/ap-384-sentry-log-enrichment-for-backend-workers
Jun 18, 2026
Merged

Sentry log enrichment for backend Workers#568
isuttell merged 3 commits into
mainfrom
codex/ap-384-sentry-log-enrichment-for-backend-workers

Conversation

@isuttell

@isuttell isuttell commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Summary

Add targeted sanitized Sentry SDK logging for backend Workers so failures carry request/component context without turning Sentry into a full log drain.

Changes

  • Enable Sentry logs with beforeSendLog and beforeSend sanitizers when SENTRY_DSN is configured.
  • Add shared worker logging helpers for structured console output, Sentry logs, and exception capture.
  • Route backend Worker onError paths and jobs op logging through the helper.
  • Route optional SENTRY_DSN to api, upload, content, jobs, stream, mcp, and apex without making deploy require it.
  • Add redaction tests for log attributes/messages, Sentry error events, and token-bearing content/agent-view paths.

Risk: HIGH

  • Areas touched: backend Workers, worker runtime, jobs logging, deploy secret routing.
  • Security: high. This touches Sentry emission and secret/token redaction; beforeSend/beforeSendLog strip banned fields and tokenized paths.
  • Performance: low expected. Info logs remain console-only; Sentry logs are warn/error/fatal only.
  • Breaking: none intended. SENTRY_DSN remains optional.

Test plan

  • pnpm --filter @agent-paste/worker-runtime build && pnpm --filter @agent-paste/worker-runtime test
  • pnpm --filter @agent-paste/content test
  • pnpm --filter @agent-paste/jobs test
  • pnpm verify
  • pnpm smoke:local
  • pnpm test:coverage
  • pre-push pnpm test:coverage:strict
  • pre-push pnpm verify
  • git diff --check
  • gitleaks protect --staged --redact

Issue: AP-384

Summary by CodeRabbit

  • New Features
    • Added standardized worker error capture and structured logging across multiple worker apps, with automatic sensitive-data redaction in both console output and Sentry reports.
    • Introduced worker logging utilities that send non-info events to Sentry when a valid DSN is available.
  • Bug Fixes
    • Unhandled worker errors are now reported in a structured way instead of plain console logging, while preserving the same internal error responses.
    • Operator logging now uses structured worker log emission.
  • Chores
    • Improved Sentry DSN provisioning coverage and expanded logging/error-handling test coverage.

@coderabbitai

coderabbitai Bot commented Jun 18, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: 234861d5-fd03-4e47-9ff8-4f550c0bb1bb

📥 Commits

Reviewing files that changed from the base of the PR and between 2d2b661 and 539d438.

📒 Files selected for processing (2)
  • packages/worker-runtime/src/logging.test.ts
  • packages/worker-runtime/src/logging.ts

📝 Walkthrough

Walkthrough

Adds a structured worker logging module (emitWorkerLog, captureWorkerError) and a Sentry event sanitizer to worker-runtime. All worker app onError handlers are migrated from console.error to captureWorkerError. The op-log.ts module in jobs is refactored to use emitWorkerLog. Sentry options gain enableLogs and beforeSend/beforeSendLog sanitization hooks. SENTRY_DSN secret routing is extended to api, upload, content, jobs, and stream apps.

Changes

Worker logging, Sentry sanitization, and app adoption

Layer / File(s) Summary
Logging module: types, emitWorkerLog, captureWorkerError, sanitization helpers
packages/worker-runtime/src/logging.ts, packages/worker-runtime/src/logging.test.ts
Defines WorkerLogLevel, WorkerLogInput, WorkerErrorLogInput; implements emitWorkerLog (structured JSON console + conditional Sentry forwarding), captureWorkerError (enriches error context, calls sentry.captureException), and sanitization helpers (sanitizeWorkerLogAttributes, sanitizeString, pathFromUrl, isSensitiveKey, normalizeKey) with token/URL redaction and resilience against logging failures. Test suite validates warn/info/error flows, attribute sanitization, and error handling.
Sentry event sanitizer and options wiring
packages/worker-runtime/src/sentry-sanitize.ts, packages/worker-runtime/src/sentry.ts, packages/worker-runtime/src/sentry.test.ts
Adds sanitizeSentryEvent with helpers for deep redaction of Sentry ErrorEvent fields (request URL, breadcrumbs, exceptions, extra). Updates sentryOptions to set enabled, enableLogs, and register beforeSend/beforeSendLog callbacks. Tests assert redaction of secrets across all event fields.
worker-runtime public exports
packages/worker-runtime/src/index.ts
Re-exports captureWorkerError, emitWorkerLog, sanitizeSentryLog, sanitizeWorkerLogAttributes, and associated types from the new logging module.
Worker app onError migration and op-log refactor
apps/api/src/index.ts, apps/content/src/index.ts, apps/jobs/src/index.ts, apps/mcp/src/index.ts, apps/upload/src/index.ts, apps/jobs/src/op-log.ts, apps/jobs/src/op-log-sentry.test.ts
Replaces console.error in all five worker app.onError handlers with captureWorkerError carrying structured component/event/environment/request metadata. Refactors op-log.ts from custom Sentry-reporting logic to emitWorkerLog calls. Updates tests to assert per-level Sentry logger spies and info-only console output.
SENTRY_DSN secret routing expansion and tests
scripts/lib/secret-routing.mjs, scripts/lib/secret-routing.test.mjs, scripts/deploy.test.mjs
Adds optional SENTRY_DSN binding to api, upload, content, jobs, and stream sections of SECRET_ROUTING. Broadens routing and deploy tests to verify the binding across all Sentry-using apps.

Sequence Diagram(s)

sequenceDiagram
  participant Worker as Worker (api/content/jobs/mcp/upload)
  participant captureWorkerError
  participant emitWorkerLog
  participant sanitizeWorkerLogAttributes
  participant Console
  participant Sentry

  rect rgba(255, 100, 100, 0.5)
    note over Worker,Sentry: Unhandled error in app.onError
    Worker->>captureWorkerError: {component, event, error, env, request, requestId}
    captureWorkerError->>sanitizeWorkerLogAttributes: error name/message + caller attributes
    sanitizeWorkerLogAttributes-->>captureWorkerError: sanitized scalar attributes
    captureWorkerError->>emitWorkerLog: level=error, merged attributes
    emitWorkerLog->>Console: JSON.stringify({level, component, event, ...attributes})
    captureWorkerError->>Sentry: captureException(error, {extra: {errorMessage}})
    emitWorkerLog->>Sentry: logger.error(event, {requestId, errorName})
  end

  rect rgba(100, 150, 255, 0.5)
    note over Worker,Sentry: Sentry beforeSend / beforeSendLog hooks
    Sentry->>Sentry: sanitizeSentryEvent(ErrorEvent)
    Sentry-->>Sentry: redacted ErrorEvent (tokens/URLs scrubbed)
  end
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related PRs

  • zaks-io/agent-paste#91: Wraps workers with Sentry.withSentry(..., sentryOptions(env)), which is the initialization counterpart to this PR's structured captureWorkerError error reporting and Sentry options sanitization hooks.
  • zaks-io/agent-paste#10: Propagates request IDs through worker app.onError handlers in the same files this PR modifies, providing the requestId parameter that captureWorkerError includes in structured error logs.
  • zaks-io/agent-paste#562: Both PRs expand SENTRY_DSN secret routing in scripts/lib/secret-routing.mjs, with the retrieved PR initially adding it for apex and this PR extending it to additional worker apps (api, upload, content, jobs, stream).

Poem

🐇 Hop hop, no more secrets in the logs!
Tokens redacted, breadcrumbs sanitized clean,
captureWorkerError leaps through the fog,
With structured JSON where console.error had been.
The rabbit guards each event with care —
No DSN left unrouted, no error laid bare! 🌿

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Sentry log enrichment for backend Workers' accurately summarizes the main change—adding Sentry SDK logging enrichment to backend Workers with structured logging and error capture.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/ap-384-sentry-log-enrichment-for-backend-workers

Comment @coderabbitai help to get the list of available commands and usage tips.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First-pass review (AP-384)

Risk: medium
Decision: Needs human review

Ticket triage

  • Intended change: Add sanitized Sentry SDK logs + structured worker logging for backend Workers when SENTRY_DSN is set, while keeping JSON console logs for Axiom and redacting secrets/tokens/URLs.
  • Scope match: Yes. Shared helpers in @agent-paste/worker-runtime, backend onError migration (api/upload/content/jobs/mcp), jobs op-log refactor, optional SENTRY_DSN routing for backend workers, and redaction tests all align with AP-384 acceptance criteria.

Review findings

Blocking: None found in the diff.

Non-blocking:

  • captureWorkerError reports the same failure via both Sentry.captureException and Sentry.logger.* — confirm this is the desired Sentry shape and won't create noisy duplicates.
  • stream now receives optional SENTRY_DSN provisioning but still has no structured captureWorkerError path (unlike api/upload/content/jobs/mcp). Likely acceptable given stream's response style, but worth a quick human sanity check.
  • CI Validate was still pending at review time (Postgres smoke/CodeQL/secret scan passed). Author checklist claims pnpm verify / coverage passed locally.

Merge checklist

Item Status
Ticket linked ✅ AP-384
Scope matches
Checks green ⚠️ Validate pending
Tests/docs appropriate ✅ Strong unit coverage for sanitizers + logging
No blocking findings
No high-risk areas ❌ See below
Merge-safe for automation

Needs human review

This PR is well-scoped and the redaction/tests look thoughtful, but it should not be auto-approved because it touches:

  1. Security-sensitive observabilitybeforeSend / beforeSendLog are the last line of defense for tokens, signed URLs, and credential material (ticket labeled risk-security-sensitive).
  2. Deploy secret routing — optional SENTRY_DSN provisioning expanded across backend workers.
  3. Cross-cutting worker runtime — shared logging/Sentry behavior affects the whole backend fleet.

Recommend a human pass focused on redaction completeness (especially real failure payloads/breadcrumbs) and confirming Sentry event shape in preview once SENTRY_DSN is configured.

Open in Web View Automation 

Sent by Cursor Automation: First Pass PR Reviewer

emitConsoleLine(level, attributes);

try {
Sentry.captureException(input.error, { extra: attributes });

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking: this path emits both captureException and sendSentryLog for the same failure. Worth confirming with Sentry that exception + structured log is intentional and won't double-count/noise alerts.

},
stream: {
STREAM_INTERNAL_SECRET: { required: true },
SENTRY_DSN: { required: false, source: "sentry" }, // Optional backend monitoring; enabled only when configured.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Non-blocking scope note: stream now gets optional SENTRY_DSN, but apps/stream wasn't migrated to captureWorkerError like the Hono workers. If stream unhandled throws should carry the same component/request context, that follow-up may belong here or a tiny stream handler wrap.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/worker-runtime/src/logging.ts`:
- Around line 49-50: The SECRET_ASSIGNMENT_PATTERN regex at line 49-50 only
matches unquoted assignment forms and misses JSON-style secret assignments with
quoted keys and values like "token":"secret". Update the
SECRET_ASSIGNMENT_PATTERN regex to handle both unquoted forms (existing pattern)
and quoted JSON-style forms where both the key and value are enclosed in double
quotes, ensuring that the pattern at line 220 will properly redact all
variations of secret assignments before they are emitted to logs and Sentry.

In `@packages/worker-runtime/src/sentry-sanitize.ts`:
- Around line 4-48: The sanitizeSentryEvent function does not handle the
event.user field, allowing unsanitized user information like email and IP
addresses to be included in the Sentry event. Add a check for event.user in the
sanitizeSentryEvent function following the existing pattern for other event
fields, and either sanitize it using sanitizeSentryRecord (or an appropriate
helper) or remove it entirely from the safe object to prevent user data leakage.

In `@packages/worker-runtime/src/sentry.test.ts`:
- Around line 75-143: The test for sanitizing Sentry error events in the
beforeSend callback is missing validation for the user field, which could
contain PII. Add a user field to the event object passed to beforeSend (e.g.,
with properties like id, email, or ip_address), then add an assertion after the
existing toMatchObject check to verify that the user field is either removed
from the returned event or properly sanitized to prevent PII leakage.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: a5d06f73-01be-4814-84c9-490b5ebc1b5b

📥 Commits

Reviewing files that changed from the base of the PR and between 5fa73f4 and 5a62ba9.

📒 Files selected for processing (16)
  • apps/api/src/index.ts
  • apps/content/src/index.ts
  • apps/jobs/src/index.ts
  • apps/jobs/src/op-log-sentry.test.ts
  • apps/jobs/src/op-log.ts
  • apps/mcp/src/index.ts
  • apps/upload/src/index.ts
  • packages/worker-runtime/src/index.ts
  • packages/worker-runtime/src/logging.test.ts
  • packages/worker-runtime/src/logging.ts
  • packages/worker-runtime/src/sentry-sanitize.ts
  • packages/worker-runtime/src/sentry.test.ts
  • packages/worker-runtime/src/sentry.ts
  • scripts/deploy.test.mjs
  • scripts/lib/secret-routing.mjs
  • scripts/lib/secret-routing.test.mjs

Comment thread packages/worker-runtime/src/logging.ts
Comment thread packages/worker-runtime/src/sentry-sanitize.ts
Comment thread packages/worker-runtime/src/sentry.test.ts

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/worker-runtime/src/logging.ts (1)

217-224: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Redact before applying the 2KB truncation.

Line 218 truncates before the redaction chain, so a secret assignment that starts near the 2048-character boundary can lose its closing delimiter and skip the Line 222/223 patterns, leaking the secret prefix.

🛡️ Proposed fix
 export function sanitizeString(value: string): string {
-  const trimmed = value.length > 2048 ? `${value.slice(0, 2048)}...[truncated]` : value;
-  return trimmed
+  const redacted = value
     .replace(API_KEY_PATTERN, "[redacted_api_key]")
     .replace(BEARER_PATTERN, "Bearer [redacted]")
     .replace(JSON_SECRET_ASSIGNMENT_PATTERN, '"$1":"[redacted]"')
     .replace(SECRET_ASSIGNMENT_PATTERN, "$1=[redacted]")
     .replace(URL_PATTERN, (match) => `[url:${pathFromUrl(match)}]`);
+  return redacted.length > 2048 ? `${redacted.slice(0, 2048)}...[truncated]` : redacted;
 }
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/worker-runtime/src/logging.ts` around lines 217 - 224, The
sanitizeString function truncates the string to 2048 characters before applying
the redaction patterns, which can break secret patterns that span the truncation
boundary and cause secrets to leak. Reverse the order of operations in the
sanitizeString function: first apply all the redaction patterns using the
replace() calls for API_KEY_PATTERN, BEARER_PATTERN,
JSON_SECRET_ASSIGNMENT_PATTERN, SECRET_ASSIGNMENT_PATTERN, and URL_PATTERN
against the original value, then apply the 2048-character truncation logic to
the redacted result.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/worker-runtime/src/logging.ts`:
- Around line 51-52: The JSON_SECRET_ASSIGNMENT_PATTERN regex uses [^"]* to
match secret values, which stops at escaped quotes within the value, leaving
part of the secret unredacted. For example, a token value like abc\"def gets
only partially redacted. Replace the [^"]* pattern in the
JSON_SECRET_ASSIGNMENT_PATTERN constant with a pattern that properly handles
escaped characters, such as (?:\\.|[^"\\])* or (?:[^"\\]|\\.)*. This ensures
that escaped quotes and other escaped characters within JSON string values are
treated as part of the value rather than terminators, so the entire secret is
redacted correctly.

---

Outside diff comments:
In `@packages/worker-runtime/src/logging.ts`:
- Around line 217-224: The sanitizeString function truncates the string to 2048
characters before applying the redaction patterns, which can break secret
patterns that span the truncation boundary and cause secrets to leak. Reverse
the order of operations in the sanitizeString function: first apply all the
redaction patterns using the replace() calls for API_KEY_PATTERN,
BEARER_PATTERN, JSON_SECRET_ASSIGNMENT_PATTERN, SECRET_ASSIGNMENT_PATTERN, and
URL_PATTERN against the original value, then apply the 2048-character truncation
logic to the redacted result.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: ecd9ebf5-0e0d-403f-9804-d557c4fbe2ab

📥 Commits

Reviewing files that changed from the base of the PR and between 5a62ba9 and 2d2b661.

📒 Files selected for processing (3)
  • packages/worker-runtime/src/logging.ts
  • packages/worker-runtime/src/sentry-sanitize.ts
  • packages/worker-runtime/src/sentry.test.ts

Comment thread packages/worker-runtime/src/logging.ts Outdated
@isuttell isuttell merged commit 78f2a98 into main Jun 18, 2026
10 checks passed
@isuttell isuttell deleted the codex/ap-384-sentry-log-enrichment-for-backend-workers branch June 18, 2026 14:26
@github-actions

Copy link
Copy Markdown

agent-paste PR preview resources were cleaned up. The shared Preview GitHub Environment is retained for future preview deploys.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant