feat(ingester): at-least-once sink delivery with bounded retry buffer by sagnik11 · Pull Request #13 · Autter-dev/autter-runtime

sagnik11 · 2026-08-17T08:52:54Z

Problem

The sink webhook forward was a single fire-and-forget fetch with a 10 s timeout and a console.warn on failure. Any consumer deploy, network blip, or slow downstream batch permanently lost the delivery — and with it the fingerprinted occurrences that feed the consumer's issue grouping and incident detection. The docs pointed at ClickHouse replay as the recovery story, but consumers couldn't retry safely and nothing implemented replay.

Change

SinkForwarder (packages/otlp-ingester/src/sink.ts): batches queue in memory and retry with exponential backoff + full jitter (1 s → 60 s, SINK_MAX_ATTEMPTS = 12 ≈ 8 min — rides out a routine consumer deploy). Network errors, timeouts, 408/429 and 5xx retry; other 4xx are consumer rejections and drop immediately. Concurrency drops to 1 while failing so recovery doesn't stampede the consumer.
Bounded buffer: SINK_MAX_BUFFERED_BATCHES = 1000 / SINK_MAX_BUFFERED_MB = 64, oldest-first eviction. Every dropped batch is logged with its signal time range for targeted ClickHouse replay.
Additive wire change: each payload now carries a unique batchId so consumers can dedupe re-deliveries (a batch can arrive twice when only the 2xx response was lost).
Observability: /healthz exposes delivery counters (sink.queued / delivered / retried / droppedOverflow / droppedPermanent / lastFailureAt …) for missed-batch monitoring; shutdown logs any undelivered batches.
Docs: new "Delivery semantics" section in ARCHITECTURE.md, compatibility table in PLAN.md, env table in the ingester README.

The durability boundary is intentional: the retry buffer is memory-only, but nothing is enqueued that wasn't already durably written to ClickHouse (ingest 503s otherwise) — ClickHouse stays the recovery source for anything the buffer cannot save.

Deploy order

Consumers whose sink writes are additive must ship their dedupe before this deploys, since retries can now legitimately deliver the same batch twice. For Autter cloud that's the monorepo's dedupe-ledger + reconciler PR (raised alongside this one).

Created with PostHog Desktop

Summary

Summary generated by Autter.
Adds at-least-once delivery for optional sink webhooks in the OTLP ingester. Fingerprinted occurrences, metric rollups, and LLM call batches are forwarded through a bounded in-memory retry buffer with exponential backoff, while ClickHouse remains the durable source for replay after buffer exhaustion or shutdown.

Changes

Added SinkForwarder in packages/otlp-ingester/src/sink.ts to:
- Deliver sink batches with a stable batchId for consumer-side deduplication.
- Retry failed deliveries with capped exponential backoff.
- Bound buffered batches and memory usage, dropping the oldest batches first when limits are exceeded.
- Expose delivery statistics for health checks and operational visibility.
Updated createIngesterApp to enqueue sink payloads after successful ClickHouse persistence and expose the forwarder on IngesterApp.
Added sink lifecycle handling to the ingester entry point so retries stop cleanly during SIGTERM/SIGINT; undelivered batches are logged as recoverable through ClickHouse replay.
Added SINK_MAX_ATTEMPTS, SINK_MAX_BUFFERED_BATCHES, and SINK_MAX_BUFFERED_MB configuration with defaults.
Updated the architecture, deployment plan, and ingester README to document at-least-once semantics, retry behavior, buffer bounds, and the new configuration.

Acceptance Criteria

When AUTTER_SINK_URL is configured, successfully persisted ingest batches are forwarded with a stable batchId.
Transient sink failures retry with capped backoff up to SINK_MAX_ATTEMPTS, without creating an unbounded in-memory queue.
Buffer limits are enforced independently by batch count and approximate memory usage, with oldest batches dropped and logged when limits are exceeded.
/healthz includes sink delivery statistics when a sink is configured and continues to report ClickHouse health correctly.
Shutdown stops new sink retries, closes the HTTP server and ClickHouse store, and logs any undelivered batches as recoverable from ClickHouse.
Ingest behavior remains unchanged when AUTTER_SINK_URL is unset.

Test Plan

Start the ingester with ClickHouse and a test sink, ingest traces, metrics, and browser data, and verify each persisted batch reaches the sink with a batchId.
Make the sink return 5xx responses, then restore it and verify retries occur with increasing delays and eventually deliver the queued batch.
Configure small SINK_MAX_ATTEMPTS, SINK_MAX_BUFFERED_BATCHES, and SINK_MAX_BUFFERED_MB values, generate failures, and verify the oldest buffered batches are dropped and logged at the configured bounds.
Query /healthz while sink deliveries are succeeding and failing, and verify the reported sink statistics reflect the pending and failed batches.
Send SIGTERM or SIGINT with pending sink work and verify shutdown logs the undelivered count and exits without preventing ClickHouse closure.
Run the existing CI build and integration workflows, including regression coverage for normalized occurrence payloads and shared RuntimeOccurrence/sink batch types.

Rollback Plan

Revert this change and redeploy the ingester image to restore direct sink forwarding behavior.
If sink delivery is unhealthy before a redeploy, unset AUTTER_SINK_URL or point it to a disabled sink endpoint; ClickHouse ingestion remains available and can serve as the replay source.
After rollback, replay any batches recorded in ClickHouse that were not delivered successfully, using the sink consumer's batchId deduplication.

Related Issues

No linked issue was identified.

^{Written for commit 26cc0c2. Summary will update on new commits.}

The sink webhook was fire-and-forget: one POST with a 10s timeout and a console.warn on failure. Any consumer deploy, network blip, or slow org-DB round trip silently lost the batch — and with it the error occurrences that feed issue grouping and incident detection downstream. Batches now queue in memory and retry with exponential backoff (1s → 60s, SINK_MAX_ATTEMPTS tries, ~8 min by default) on network errors, timeouts, 408/429 and 5xx; other 4xx drop immediately. The buffer is bounded (SINK_MAX_BUFFERED_BATCHES / SINK_MAX_BUFFERED_MB) with oldest-first eviction, and every drop is logged with its signal time range so the consumer can replay it from ClickHouse — which stays the recovery source: nothing is enqueued that wasn't already durably written there. Each payload additively gains a unique batchId so consumers can dedupe re-deliveries (a batch can arrive twice when only the 2xx was lost). /healthz now exposes sink delivery counters for missed-batch monitoring, and shutdown logs any undelivered batches. Generated-By: PostHog Desktop Task-Id: 97b59a29-5efe-4bce-ab15-fec99be87fe5

autter-dev

🔴 Autter review in progress — running security, correctness & dependency checks on this PR. Follow live step-by-step progress on the autter/review-gate check in the merge box. Merge is blocked until the gate completes; Autter approves automatically when the review comes back clean, and releases this hold with a neutral review when it finds non-blocking issues.

autter-dev

Autter posted 1 finding(s) as review threads below (🟠 1). Each carries a copy-paste AI fix prompt.

autter-dev · 2026-08-17T08:58:54Z

+			// batch — retrying the same body cannot succeed.
+			permanent = res.status < 500 && res.status !== 408 && res.status !== 429;
+		} catch (err) {
+			failure = err instanceof Error ? err.message : String(err);


🟠 Raw sink exception messages exposed through unauthenticated health checks — Risk: 62/100

When fetchImpl rejects, SinkForwarder.send() stores the raw exception message in lastFailureMessage. GET /healthz returns sink.stats(), exposing those messages to unauthenticated health clients. Store a sanitized failure category/message instead of the raw exception text.

🛠 AI fix prompt (copy & paste into your coding agent)

Do not expose the raw error message in SinkStats or GET /healthz. Store it only in server-side logs, or map failures to a fixed safe category such as "timeout", "connection_error", or "http_error" before returning the health response.

_{Flagged by Autter security & observability checks.}

autter-dev

Autter posted 1 finding(s) as review threads below (🟡 1). Each carries a copy-paste AI fix prompt.

autter-dev · 2026-08-17T09:00:19Z

+	) {}
+
+	/** Queue a batch for delivery. No-op when there is nothing to send. */
+	enqueue(


🟡 [ai] Split batch shaping from queue mutation — Risk: 35/100

enqueue combines eligibility checks, payload shaping and serialization, signal-range calculation, queue mutation, buffer-bound enforcement, and delivery scheduling. Extracting batch construction from queue management would allow retry-buffer behavior and payload shaping to be tested independently; the current code has no functional validation defect.

🛠 AI fix prompt (copy & paste into your coding agent)

Extract pure helpers for building the serialized batch and calculating its signal range, then keep enqueue focused on queue mutation, bounds enforcement, and pump scheduling.

_{Flagged by Autter security & observability checks.}

autter-dev · 2026-08-17T09:00:25Z

🚦 Pre-merge checks · ⚠️ 15 warning, ✅ 154 passed

Needs attention

Check	Status	Explanation
Removed observability	⚠️ Warning	1 potential issue(s) detected (max risk 65/100): packages/otlp-ingester/src/server.ts:208.
Silent exception swallowing	⚠️ Warning	1 potential issue(s) detected (max risk 70/100): packages/otlp-ingester/src/sink.ts:247.
Stack trace leakage	⚠️ Warning	1 potential issue(s) detected (max risk 62/100): packages/otlp-ingester/src/sink.ts:258.
Idempotency key not detected	⚠️ Warning	1 potential issue(s) detected (max risk 48/100): packages/otlp-ingester/src/sink.ts:5.
Possible non-atomic read-modify-write	⚠️ Warning	1 potential issue(s) detected (max risk 45/100): packages/otlp-ingester/src/sink.ts:247.
Batch size limit not detected	⚠️ Warning	1 potential issue(s) detected (max risk 48/100): packages/otlp-ingester/src/sink.ts:57.
Missing linked tracker issue	⚠️ Warning	2 potential issue(s) detected (max risk 50/100): packages/otlp-ingester/src/sink.ts:1, packages/otlp-ingester/src/server.ts:41.
Missing CODEOWNERS reviewer approval	⚠️ Warning	1 potential issue(s) detected (max risk 65/100): packages/otlp-ingester/src/config.ts:26.
Missing security-team review on sensitive path	⚠️ Warning	1 potential issue(s) detected (max risk 60/100): packages/otlp-ingester/src/config.ts:82.
Source changes without matching tests	⚠️ Warning	4 potential issue(s) detected (max risk 85/100): packages/otlp-ingester/src/index.ts:5, packages/otlp-ingester/src/server.ts:22, packages/otlp-ingester/src/sink.ts:1, packages/otlp-ingester/src/server.ts:41.
Hallucinated import (package not installed)	⚠️ Warning	1 potential issue(s) detected (max risk 80/100): packages/otlp-ingester/src/sink.ts:3.
Runtime error risk	⚠️ Warning	1 finding(s) on changed lines.
Excessive complexity	⚠️ Warning	2 finding(s) on changed lines.
Dead export (no callers)	⚠️ Warning	1 finding(s) on changed lines.
Complexity Guard	⚠️ Warning	1 finding(s) on changed lines.

✅ Passed checks (154)

Check	Status	Explanation
Too many files changed	✅ Passed	Changed 7 file(s), within the limit of 50.
Too many lines changed	✅ Passed	Changed 435 line(s), within the limit of 1000.
Too many unrelated chapters	✅ Passed	4 chapter(s) detected, within the limit of 6.
Generated files hiding real changes	✅ Passed	Generated-file volume (0 lines) does not obscure the 435 hand-written line(s).
Missing PR context	✅ Passed	PR context looks sufficient.
Mixed concerns (refactor + behavior change)	✅ Passed	The PR is a focused behavioral reliability change introducing at-least-once sink delivery, retries, buffering, and lifecycle handling; it does not combine a separate no-behavior refactor with that change.
Migration + app logic + UI combined in one PR	✅ Passed	The PR contains no database migration files and no user-interface changes; it only changes ingester behavior, configuration, and documentation.
Sensitive data in logs	✅ Passed	No sensitive data in logs issues detected.
Log injection	✅ Passed	No log injection issues detected.
Missing audit logging	✅ Passed	No missing audit logging issues detected.
Unhandled promise rejection	✅ Passed	No unhandled promise rejection issues detected.
Circuit breaker not detected	✅ Passed	No circuit breaker not detected issues detected.
Multi-write without detected transaction	✅ Passed	No multi-write without detected transaction issues detected.
Possible TOCTOU in critical path	✅ Passed	No possible toctou in critical path issues detected.
Optimistic locking not detected	✅ Passed	No optimistic locking not detected issues detected.
Rate limiting not detected	✅ Passed	No rate limiting not detected issues detected.
Rate limiting removed	✅ Passed	No rate limiting removed issues detected.
Pagination not detected	✅ Passed	No pagination not detected issues detected.
Publicly exposed storage	✅ Passed	No publicly exposed storage issues detected.
Over-permissive IAM policy	✅ Passed	No over-permissive iam policy issues detected.
Security group open to the internet	✅ Passed	No security group open to the internet issues detected.
Unencrypted storage at rest	✅ Passed	No unencrypted storage at rest issues detected.
Infrastructure missing access logging	✅ Passed	No infrastructure missing access logging issues detected.
Hardcoded secret in IaC	✅ Passed	No hardcoded secret in iac issues detected.
Infrastructure misconfiguration	✅ Passed	No infrastructure misconfiguration issues detected.
Deprecated Kubernetes API version	✅ Passed	No deprecated kubernetes api version issues detected.
Compound IaC attack chain	✅ Passed	No compound iac attack chain issues detected.
Prompt injection risk	✅ Passed	No LLM/AI-integration code touched by this diff.
LLM output used in a dangerous sink	✅ Passed	No LLM/AI-integration code touched by this diff.
Sensitive data in prompt or system-prompt leakage	✅ Passed	No LLM/AI-integration code touched by this diff.
Over-privileged LLM tool / excessive agency	✅ Passed	No LLM/AI-integration code touched by this diff.
Missing validation on an LLM-driven decision	✅ Passed	No LLM/AI-integration code touched by this diff.
Unbounded LLM usage (denial-of-wallet)	✅ Passed	No LLM/AI-integration code touched by this diff.
Table exposed without row-level security	✅ Passed	No row-level-security-related code touched by this diff.
Over-broad row-level security policy	✅ Passed	No row-level-security-related code touched by this diff.
Code path that bypasses row-level security	✅ Passed	No row-level-security-related code touched by this diff.
Privileged database credential reachable from the client	✅ Passed	No row-level-security-related code touched by this diff.
Privileged query without row-level scoping	✅ Passed	No row-level-security-related code touched by this diff.
Template-default gradient styling	✅ Passed	No added frontend pages or design-slop markers in this diff.
Interchangeable AI marketing copy	✅ Passed	No added frontend pages or design-slop markers in this diff.
Placeholder content shipped to users	✅ Passed	No added frontend pages or design-slop markers in this diff.
Emoji standing in for an icon system	✅ Passed	No added frontend pages or design-slop markers in this diff.
Call-to-action that goes nowhere	✅ Passed	No added frontend pages or design-slop markers in this diff.
Templated page composition	✅ Passed	No added frontend pages or design-slop markers in this diff.
Merge-blocking marker left in the change	✅ Passed	No pending-work markers added by this diff.
Known-defect marker shipped in code	✅ Passed	No pending-work markers added by this diff.
Untracked TODO without an issue reference	✅ Passed	No pending-work markers added by this diff.
Test disabled or left pending	✅ Passed	No pending-work markers added by this diff.
PII in logs	✅ Passed	No pii in logs issues detected.
PII or internals leaked in error response	✅ Passed	No pii or internals leaked in error response issues detected.
PII stored without application-level encryption	✅ Passed	No pii stored without application-level encryption issues detected.
User data stored without retention controls	✅ Passed	No user data stored without retention controls issues detected.
PII sent to external / cross-border destination	✅ Passed	No pii sent to external / cross-border destination issues detected.
Lockfile resolution / integrity tampered	✅ Passed	No lockfile resolution / integrity tampered issues detected.
Dependency runs install-time lifecycle script	✅ Passed	No dependency runs install-time lifecycle script issues detected.
Possible dependency-confusion attack	✅ Passed	No possible dependency-confusion attack issues detected.
Lockfile resolves a dependency the manifest does not declare	✅ Passed	No lockfile resolves a dependency the manifest does not declare issues detected.
Checked-in build artefact modified without source change	✅ Passed	No checked-in build artefact modified without source change issues detected.
Dockerfile build-step is insecure	✅ Passed	No dockerfile build-step is insecure issues detected.
External artefact pulled in without integrity pinning	✅ Passed	No external artefact pulled in without integrity pinning issues detected.
Changed export, importer not updated	✅ Passed	No changed export with an un-updated importer detected.
Migration missing rollback / down step	✅ Passed	No migration missing rollback / down step issues detected.
Frontend importing database client directly	✅ Passed	No frontend importing database client directly issues detected.
Route handler bypassing service layer	✅ Passed	No route handler bypassing service layer issues detected.
Backend service importing UI module	✅ Passed	No backend service importing ui module issues detected.
Cross-context internals import	✅ Passed	No cross-context internals import issues detected.
Workspace package rule violation	✅ Passed	No workspace package rule violation issues detected.
Inconsistent logging pattern	✅ Passed	No inconsistent logging pattern issues detected.
Inconsistent error handling	✅ Passed	No inconsistent error handling issues detected.
Endpoint missing input validation	✅ Passed	No endpoint missing input validation issues detected.
Multi-write without transaction wrapper	✅ Passed	No multi-write without transaction wrapper issues detected.
New feature shipped without feature flag	✅ Passed	No new feature shipped without feature flag issues detected.
Module placed in the wrong workspace package	✅ Passed	No module placed in the wrong workspace package issues detected.
Direct env-var access bypasses config module	✅ Passed	No direct env-var access bypasses config module issues detected.
Nonexistent package (not found in registry)	✅ Passed	No nonexistent package (not found in registry) issues detected.
Call to function that does not exist	✅ Passed	No call to function that does not exist issues detected.
Generic placeholder identifier in production logic	✅ Passed	No generic placeholder identifier in production logic issues detected.
Repetitive boilerplate (duplicated block)	✅ Passed	No repetitive boilerplate (duplicated block) issues detected.
Overbroad try/catch swallowing all exceptions	✅ Passed	No overbroad try/catch swallowing all exceptions issues detected.
TODO / FIXME on critical path	✅ Passed	No todo / fixme on critical path issues detected.
Comment contradicts or fabricates code behaviour	✅ Passed	No comment contradicts or fabricates code behaviour issues detected.
Abstraction defined but never used	✅ Passed	No abstraction defined but never used issues detected.
Code style differs from rest of codebase	✅ Passed	No code style differs from rest of codebase issues detected.
Established pattern ignored	✅ Passed	No established pattern ignored issues detected.
Unhandled edge case (null / empty / zero / boundary)	✅ Passed	No unhandled edge case (null / empty / zero / boundary) issues detected.
Doc-copy code with insecure defaults	✅ Passed	No doc-copy code with insecure defaults issues detected.
Dead code (defined but never referenced)	✅ Passed	No dead code (defined but never referenced) issues detected.
Deprecated API call	✅ Passed	No deprecated api call issues detected.
API pattern from wrong library version	✅ Passed	No api pattern from wrong library version issues detected.
API endpoint removed	✅ Passed	No api endpoint removed issues detected.
HTTP method changed (GET ↔ POST etc.)	✅ Passed	No http method changed (get ↔ post etc.) issues detected.
New required field added to request	✅ Passed	No new required field added to request issues detected.
Field removed from response schema	✅ Passed	No field removed from response schema issues detected.
Response field type changed	✅ Passed	No response field type changed issues detected.
HTTP status code changed	✅ Passed	No http status code changed issues detected.
Auth requirement added / removed / changed	✅ Passed	No auth requirement added / removed / changed issues detected.
Error response shape changed	✅ Passed	No error response shape changed issues detected.
Pagination behaviour changed	✅ Passed	No pagination behaviour changed issues detected.
Outbound webhook payload schema changed	✅ Passed	No outbound webhook payload schema changed issues detected.
GraphQL field removed without deprecation	✅ Passed	No graphql field removed without deprecation issues detected.
GraphQL enum value removed	✅ Passed	No graphql enum value removed issues detected.
SQL injection	✅ Passed	No sql injection issues detected.
Cross-site scripting (XSS)	✅ Passed	No cross-site scripting (xss) issues detected.
Path traversal	✅ Passed	No path traversal issues detected.
Command injection	✅ Passed	No command injection issues detected.
Insecure deserialization	✅ Passed	No insecure deserialization issues detected.
Weak cryptography	✅ Passed	No weak cryptography issues detected.
Hardcoded secret	✅ Passed	No hardcoded secret issues detected.
Insecure randomness for security material	✅ Passed	No insecure randomness for security material issues detected.
Unsafe file upload	✅ Passed	No unsafe file upload issues detected.
Missing input validation	✅ Passed	No missing input validation issues detected.
Unsafe CORS configuration	✅ Passed	No unsafe cors configuration issues detected.
Unsafe / open redirect	✅ Passed	No unsafe / open redirect issues detected.
Missing CSRF protection	✅ Passed	No missing csrf protection issues detected.
Unsafe cookie / session settings	✅ Passed	No unsafe cookie / session settings issues detected.
Sensitive data exposure	✅ Passed	No sensitive data exposure issues detected.
API key in source	✅ Passed	No api key in source detected.
Access token in source	✅ Passed	No access token in source detected.
Private key in source	✅ Passed	No private key in source detected.
Database connection URL with embedded credentials	✅ Passed	No database connection url with embedded credentials detected.
Cloud credential in source	✅ Passed	No cloud credential in source detected.
Webhook signing secret in source	✅ Passed	No webhook signing secret in source detected.
OAuth client secret in source	✅ Passed	No oauth client secret in source detected.
JWT signing secret in source	✅ Passed	No jwt signing secret in source detected.
Hardcoded password	✅ Passed	No hardcoded password detected.
Auth middleware removed from route	✅ Passed	No auth middleware removed from route issues detected.
Route protection changed (protected → public)	✅ Passed	No route protection changed (protected → public) issues detected.
Permission / RBAC check removed	✅ Passed	No permission / rbac check removed issues detected.
Required role weakened	✅ Passed	No required role weakened issues detected.
Admin-only route exposed to lower privilege	✅ Passed	No admin-only route exposed to lower privilege issues detected.
Token validation skipped in middleware chain	✅ Passed	No token validation skipped in middleware chain issues detected.
JWT verification weakened or changed	✅ Passed	No jwt verification weakened or changed issues detected.
Session expiration / TTL changed	✅ Passed	No session expiration / ttl changed issues detected.
Password reset flow changed	✅ Passed	No password reset flow changed issues detected.
OAuth callback / redirect handling changed	✅ Passed	No oauth callback / redirect handling changed issues detected.
Webhook endpoint missing signature verification	✅ Passed	No webhook endpoint missing signature verification issues detected.
Public route touches private/PII data	✅ Passed	No public route touches private/pii data issues detected.
Frontend performance issue	✅ Passed	No additional explanation was reported.
Frontend security issue	✅ Passed	No additional explanation was reported.
Frontend correctness issue	✅ Passed	No additional explanation was reported.
Accessibility issue	✅ Passed	No additional explanation was reported.
Frontend maintainability issue	✅ Passed	No additional explanation was reported.
Code correctness issue	✅ Passed	No additional explanation was reported.
Resource leak risk	✅ Passed	No additional explanation was reported.
Data integrity risk	✅ Passed	No additional explanation was reported.
Maintainability issue	✅ Passed	No additional explanation was reported.
Co-change coupling	✅ Passed	No additional explanation was reported.
Redundant alias / duplicate import	✅ Passed	No additional explanation was reported.
Redundant type construct	✅ Passed	No additional explanation was reported.
Simplifiable code	✅ Passed	No additional explanation was reported.
Unnecessary type assertion	✅ Passed	No additional explanation was reported.
Module smell	✅ Passed	No additional explanation was reported.
Code duplication / DRY violation	✅ Passed	No additional explanation was reported.
Bundle Size Monitor	✅ Passed	No additional explanation was reported.

_{This comment is updated automatically whenever Autter reviews a new PR revision.}

autter-dev · 2026-08-17T09:00:28Z

🧭 PR hygiene & process suggestions

Autter has 8 suggestion(s) about the shape of this PR (size, scope, reviewability). These are process guidance — not code defects — so they are consolidated here instead of posted as inline comments on individual files.

🟠 Source changes without matching tests — Risk: 75/100

This source file was changed but no sibling test file is added or modified anywhere in the PR. Blast radius — skipping this guardrail cascades to the downstream usage that depends on this file: functions shutdown, IngesterConfig, IngesterApp, createIngesterApp, loadConfig; scopes @autter/otlp-ingester; dependent files packages/otlp-ingester/src/config.ts, packages/otlp-ingester/src/server.ts, ./server.js, ./config.js, ./types.js.