Skip to content

feat(trigger-service): reserve-before-dispatch dedup (TS-IMPL-009)#662

Merged
toddysm merged 2 commits into
mainfrom
ts-impl-009-dedup
Jun 5, 2026
Merged

feat(trigger-service): reserve-before-dispatch dedup (TS-IMPL-009)#662
toddysm merged 2 commits into
mainfrom
ts-impl-009-dedup

Conversation

@toddysm
Copy link
Copy Markdown
Owner

@toddysm toddysm commented Jun 5, 2026

Summary

Implements TS-IMPL-009 — Dedup / idempotency (#639).

Adds custos_trigger.dedup, which deduplicates inbound events on
hash(subscriptionId, source.eventId) using the SPL put_dedup_key
reserve-or-read primitive.

What's included

  • compute_dedup_key(subscription_id, event_id) — deterministic,
    length-prefixed SHA-256 key (no delimiter aliasing) namespaced under
    trigger.dedup.v1. Stable across replicas/restarts, which is what makes the
    dedup window correct under at-least-once redelivery.
  • Deduplicator.reserve(...) — calls put_dedup_key and maps
    DedupReservedUNSEEN / DedupDuplicateDUPLICATE. TTL defaults to
    TRIGGER_DEDUP_TTL_SECONDS with a per-call override.
  • Deduplicator.guard(...) — reserve-before-dispatch async context
    manager. On the unseen path the reservation stands once the body completes;
    if the body raises (dispatch failed) the reservation is rolled back so
    the redelivery re-attempts instead of being suppressed as a false duplicate
    — i.e. the dedup key is not committed on dispatch failure
    (design § Failure Modes, row 1).
  • InMemoryTriggerMetadataStore.release_dedup_key — backs the rollback.
    SPL exposes no selective dedup-clear in v1 (design § TODO-007), so the
    rollback is best-effort: it deletes the key when the store advertises the
    hook (in-process backend) and falls back to TTL expiry otherwise (Postgres
    v1). The hard guarantee remains the Workflow Service StartRun
    idempotencyKey; this store is the fast-path duplicate suppressor.

Acceptance criteria

  • First event reserves and returns unseen.
  • Replay within window returns duplicate and suppresses dispatch.
  • Key not committed when dispatch fails (design § Failure Modes row 1).
  • Coverage ≥ 90% (dedup.py 100%; suite 99.84%).
  • ruff + mypy clean.

Closes #639

Add `custos_trigger.dedup` deduplicating inbound events on
`hash(subscriptionId, source.eventId)` over the SPL `put_dedup_key`
reserve-or-read primitive.

- `compute_dedup_key`: deterministic, length-prefixed SHA-256 key
  (no delimiter aliasing) namespaced under `trigger.dedup.v1`.
- `Deduplicator.reserve`: maps `DedupReserved`->UNSEEN / `DedupDuplicate`
  ->DUPLICATE; TTL from `TRIGGER_DEDUP_TTL_SECONDS` with per-call override.
- `Deduplicator.guard`: reserve-before-dispatch context manager that rolls
  back the reservation when the guarded dispatch raises, so the dedup key is
  not committed on dispatch failure and the redelivery re-attempts
  (design § Failure Modes row 1). Rollback is best-effort: it deletes the
  key when the store advertises `release_dedup_key` (in-process backend) and
  falls back to TTL expiry otherwise (Postgres v1; design § TODO-007).
- Add `InMemoryTriggerMetadataStore.release_dedup_key` to back the rollback.

Closes #639
Copilot AI review requested due to automatic review settings June 5, 2026 03:24
@toddysm toddysm added type:implementation Implementation work item phase:implementation Implementation phase component:trigger-service Trigger Service component labels Jun 5, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a Trigger Service deduplication helper (TS-IMPL-009) that reserves a deterministic dedup key via SPL put_dedup_key and provides an async guard to suppress replays and roll back reservations on dispatch failure.

Changes:

  • Introduces custos_trigger.dedup with deterministic key derivation, Deduplicator.reserve(), and Deduplicator.guard() rollback behavior.
  • Extends the in-memory Trigger metadata store with a best-effort release_dedup_key hook to support rollback in dev/tests.
  • Adds unit tests covering key stability, TTL behavior, duplicate detection, and rollback semantics; marks TS-IMPL-009 complete in the design TODO list.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
src/services/trigger-service/src/custos_trigger/dedup.py New dedup module: SHA-256 key derivation, reserve/guard APIs, and best-effort rollback support.
src/services/trigger-service/src/custos_trigger/providers.py Adds InMemoryTriggerMetadataStore.release_dedup_key to support rollback of dedup reservations.
src/services/trigger-service/tests/test_dedup.py New test suite validating dedup key behavior, TTL expiry/override, duplicate suppression, and rollback on dispatch failure.
design/components/trigger-service/todos.md Marks TS-IMPL-009 as completed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/services/trigger-service/src/custos_trigger/dedup.py Outdated
Comment thread src/services/trigger-service/src/custos_trigger/dedup.py
Comment thread src/services/trigger-service/src/custos_trigger/dedup.py
…ack errors

Address Copilot review on PR #662:
- reserve() now matches DedupReserved/DedupDuplicate explicitly and raises
  TypeError on unexpected put_dedup_key result types (fail fast on store
  contract drift instead of silently misclassifying as duplicate).
- guard() rollback wraps release() in contextlib.suppress so a best-effort
  release failure never masks the real dispatch exception.
@toddysm toddysm merged commit a415de1 into main Jun 5, 2026
30 checks passed
@toddysm toddysm deleted the ts-impl-009-dedup branch June 5, 2026 03:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:trigger-service Trigger Service component phase:implementation Implementation phase type:implementation Implementation work item

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TS-IMPL-009: Dedup / idempotency

2 participants