Skip to content

feat(trigger-service): add pipeline dispatcher (TS-IMPL-014)#667

Merged
toddysm merged 2 commits into
mainfrom
ts-impl-014-dispatcher
Jun 5, 2026
Merged

feat(trigger-service): add pipeline dispatcher (TS-IMPL-014)#667
toddysm merged 2 commits into
mainfrom
ts-impl-014-dispatcher

Conversation

@toddysm
Copy link
Copy Markdown
Owner

@toddysm toddysm commented Jun 5, 2026

Summary

Implements TS-IMPL-014 — the Dispatcher, the tail of the linear pipeline
(Classify → Match → Dedup → Dispatch). It turns matches into Workflow Service
Internal RPCs with retry, dead-letter, dedup-commit ordering, and fan-out loop
protection.

Closes #644

What's included

  • custos_trigger/pipeline/dispatch.pyDispatcher:
    • start matchStartRun(workflowVersionId, inputs) via the
      WorkflowServiceClient; resume match
      RaiseExternalEvent(runId, stepId, eventName, payload).
    • Retry then dead-letter — transient failures (408/429/5xx,
      transport) are retried with exponential backoff up to
      TRIGGER_DISPATCH_MAX_RETRIES; permanent failures and exhausted budgets
      dead-letter with a trigger.dispatch.failed audit event.
    • Dedup commits only after a confirmed dispatch — the call runs inside
      Deduplicator.guard, so the dedup key is rolled back on failure and the
      redelivery can re-attempt (design § Failure Modes row 1).
    • Fan-out loop guard — a dispatch whose inbound event chain depth
      exceeds the per-tenant TRIGGER_FANOUT_MAX_DEPTH limit is rejected and
      trigger.loop.detected is audited.
    • Every RPC carries the deterministic dedup key as its idempotencyKey for
      the hard idempotency guarantee.
    • AuditSink protocol + NoopAuditSink default (real OTel/audit lands in
      TS-IMPL-019); DispatchOutcome / DispatchStatus result types.
  • settings.py — adds the TRIGGER_FANOUT_MAX_DEPTH knob (default 16).
  • pipeline/__init__.py — exports the dispatcher surface.
  • Teststests/test_dispatch.py covering both arms, duplicate skip,
    retry→success, retry-exhaustion→dead-letter, non-retryable→immediate
    dead-letter, dedup rollback on failure, loop rejection + boundary, and the
    Noop audit/default-sink paths (100% on dispatch.py); plus settings coverage
    for the new knob.

Notes

  • inputMapping ${{ … }} placeholder resolution is performed by the receiver
    before handing the subscription to the dispatcher; the dispatcher forwards the
    resolved input_mapping as the start inputs.
  • Candidate enumeration stays with the receiver/caller (the locked SPL store has
    no list surface), matching the TS-IMPL-012 matcher design.

Acceptance

  • Both dispatch arms covered; retry then dead-letter on persistent failure.
  • Dedup key committed only after a confirmed dispatch; loop-depth limit rejects + audits.
  • ruff / ruff format / mypy clean; pytest ≥90% (99.9% total, dispatch.py 100%).

Add the dispatcher that turns start/resume matches into Workflow Service
StartRun / RaiseExternalEvent calls. Transient failures retry with exponential
backoff up to TRIGGER_DISPATCH_MAX_RETRIES; permanent failures and exhausted
budgets dead-letter with a trigger.dispatch.failed audit. The dispatch runs
inside the dedup guard so the dedup key commits only after a confirmed dispatch
and rolls back on failure. A per-tenant fan-out depth limit
(TRIGGER_FANOUT_MAX_DEPTH) rejects looping chains and audits
trigger.loop.detected. Every RPC carries the deterministic dedup key as its
idempotencyKey. Adds an AuditSink protocol with a Noop default until TS-IMPL-019.

Closes #644
Copilot AI review requested due to automatic review settings June 5, 2026 04:57
@toddysm toddysm added type:implementation Implementation work item phase:implementation Implementation phase component:trigger-service Trigger Service component labels Jun 5, 2026
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Implements the Trigger Service “Dispatcher” (TS-IMPL-014), the final stage of the trigger pipeline that turns matched subscriptions into Workflow Service internal RPC calls with dedup-commit ordering, bounded retries, dead-lettering, and fan-out loop protection.

Changes:

  • Adds Dispatcher with start/resume dispatch arms, exponential backoff retry, dead-letter outcomes, and depth-limit loop rejection + audit sink surface.
  • Introduces TRIGGER_FANOUT_MAX_DEPTH setting (default 16) and exports dispatcher symbols from custos_trigger.pipeline.
  • Adds comprehensive dispatcher unit tests plus settings coverage for the new knob; marks TS-IMPL-014 complete in the trigger-service design TODOs.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/services/trigger-service/src/custos_trigger/pipeline/dispatch.py New dispatcher implementation with retry/dead-letter/dedup-guard ordering and loop-depth guard + auditing.
src/services/trigger-service/src/custos_trigger/settings.py Adds TRIGGER_FANOUT_MAX_DEPTH configuration wiring and default.
src/services/trigger-service/src/custos_trigger/pipeline/init.py Exports dispatcher types/surface from the pipeline package and updates module docstring.
src/services/trigger-service/tests/test_dispatch.py New test suite covering start/resume dispatch, retry/dead-letter behavior, dedup rollback, and loop guard.
src/services/trigger-service/tests/test_settings.py Extends settings tests to include the new fanout depth knob.
design/components/trigger-service/todos.md Marks TS-IMPL-014 as completed.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/services/trigger-service/src/custos_trigger/settings.py
… + docs

Wire the new fan-out depth limit into the trigger-service Helm chart
(values.fanoutMaxDepth, ConfigMap TRIGGER_FANOUT_MAX_DEPTH), document it in
design.md Configuration + Failure Modes, and add it to the chart README env
table so operators can tune the loop guard in deployments. Extends the helm
render test to assert the rendered default.
@toddysm toddysm merged commit 170b03d into main Jun 5, 2026
36 checks passed
@toddysm toddysm deleted the ts-impl-014-dispatcher branch June 5, 2026 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

component:trigger-service Trigger Service component phase:implementation Implementation phase type:implementation Implementation work item

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants