feat(trigger-service): reserve-before-dispatch dedup (TS-IMPL-009)#662
Merged
Conversation
Add `custos_trigger.dedup` deduplicating inbound events on `hash(subscriptionId, source.eventId)` over the SPL `put_dedup_key` reserve-or-read primitive. - `compute_dedup_key`: deterministic, length-prefixed SHA-256 key (no delimiter aliasing) namespaced under `trigger.dedup.v1`. - `Deduplicator.reserve`: maps `DedupReserved`->UNSEEN / `DedupDuplicate` ->DUPLICATE; TTL from `TRIGGER_DEDUP_TTL_SECONDS` with per-call override. - `Deduplicator.guard`: reserve-before-dispatch context manager that rolls back the reservation when the guarded dispatch raises, so the dedup key is not committed on dispatch failure and the redelivery re-attempts (design § Failure Modes row 1). Rollback is best-effort: it deletes the key when the store advertises `release_dedup_key` (in-process backend) and falls back to TTL expiry otherwise (Postgres v1; design § TODO-007). - Add `InMemoryTriggerMetadataStore.release_dedup_key` to back the rollback. Closes #639
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a Trigger Service deduplication helper (TS-IMPL-009) that reserves a deterministic dedup key via SPL put_dedup_key and provides an async guard to suppress replays and roll back reservations on dispatch failure.
Changes:
- Introduces
custos_trigger.dedupwith deterministic key derivation,Deduplicator.reserve(), andDeduplicator.guard()rollback behavior. - Extends the in-memory Trigger metadata store with a best-effort
release_dedup_keyhook to support rollback in dev/tests. - Adds unit tests covering key stability, TTL behavior, duplicate detection, and rollback semantics; marks TS-IMPL-009 complete in the design TODO list.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/services/trigger-service/src/custos_trigger/dedup.py | New dedup module: SHA-256 key derivation, reserve/guard APIs, and best-effort rollback support. |
| src/services/trigger-service/src/custos_trigger/providers.py | Adds InMemoryTriggerMetadataStore.release_dedup_key to support rollback of dedup reservations. |
| src/services/trigger-service/tests/test_dedup.py | New test suite validating dedup key behavior, TTL expiry/override, duplicate suppression, and rollback on dispatch failure. |
| design/components/trigger-service/todos.md | Marks TS-IMPL-009 as completed. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…ack errors Address Copilot review on PR #662: - reserve() now matches DedupReserved/DedupDuplicate explicitly and raises TypeError on unexpected put_dedup_key result types (fail fast on store contract drift instead of silently misclassifying as duplicate). - guard() rollback wraps release() in contextlib.suppress so a best-effort release failure never masks the real dispatch exception.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements TS-IMPL-009 — Dedup / idempotency (#639).
Adds
custos_trigger.dedup, which deduplicates inbound events onhash(subscriptionId, source.eventId)using the SPLput_dedup_keyreserve-or-read primitive.
What's included
compute_dedup_key(subscription_id, event_id)— deterministic,length-prefixed SHA-256 key (no delimiter aliasing) namespaced under
trigger.dedup.v1. Stable across replicas/restarts, which is what makes thededup window correct under at-least-once redelivery.
Deduplicator.reserve(...)— callsput_dedup_keyand mapsDedupReserved→UNSEEN/DedupDuplicate→DUPLICATE. TTL defaults toTRIGGER_DEDUP_TTL_SECONDSwith a per-call override.Deduplicator.guard(...)— reserve-before-dispatch async contextmanager. On the unseen path the reservation stands once the body completes;
if the body raises (dispatch failed) the reservation is rolled back so
the redelivery re-attempts instead of being suppressed as a false duplicate
— i.e. the dedup key is not committed on dispatch failure
(design § Failure Modes, row 1).
InMemoryTriggerMetadataStore.release_dedup_key— backs the rollback.SPL exposes no selective dedup-clear in v1 (design § TODO-007), so the
rollback is best-effort: it deletes the key when the store advertises the
hook (in-process backend) and falls back to TTL expiry otherwise (Postgres
v1). The hard guarantee remains the Workflow Service
StartRunidempotencyKey; this store is the fast-path duplicate suppressor.Acceptance criteria
unseen.duplicateand suppresses dispatch.dedup.py100%; suite 99.84%).ruff+mypyclean.Closes #639