From 81ace35c4ef492f395f70d2f15f60fc18865b63f Mon Sep 17 00:00:00 2001 From: Juan Pablo Vega Date: Thu, 18 Jun 2026 16:37:01 +0200 Subject: [PATCH 1/4] [docs] Plan gateway triggers: research, proposal, and WP/WL/WS breakdown Adds the gateway-triggers design set under docs/designs/gateway-triggers/: research, proposal, gap, mimics, mapping, plan, and the wp/ specs + status trackers + lane/stream runbook. Triggers are the inbound dual of webhooks: Composio provider events invoke Agenta workflows. plan.md frames the build as three views over seven units: Work Packages (functional DAG, fan-in), Work Lanes (GitButler merge tree, no fan-in), Work Streams (parallel subagent assignments against frozen contracts). No application code. WP0 notes the one migration subtlety: the connection rename lands in the shared core_oss chain, not the parked core tree. Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/designs/gateway-triggers/gap.md | 140 ++++++ docs/designs/gateway-triggers/mapping.md | 330 ++++++++++++++ docs/designs/gateway-triggers/mimics.md | 307 +++++++++++++ docs/designs/gateway-triggers/plan.md | 409 ++++++++++++++++++ docs/designs/gateway-triggers/proposal.md | 236 ++++++++++ docs/designs/gateway-triggers/research.md | 403 +++++++++++++++++ .../designs/gateway-triggers/wp/WL-runbook.md | 156 +++++++ docs/designs/gateway-triggers/wp/WP0-specs.md | 104 +++++ .../designs/gateway-triggers/wp/WP0-status.md | 65 +++ docs/designs/gateway-triggers/wp/WP1-specs.md | 66 +++ .../designs/gateway-triggers/wp/WP1-status.md | 43 ++ docs/designs/gateway-triggers/wp/WP2-specs.md | 51 +++ .../designs/gateway-triggers/wp/WP2-status.md | 43 ++ docs/designs/gateway-triggers/wp/WP3-specs.md | 64 +++ .../designs/gateway-triggers/wp/WP3-status.md | 33 ++ docs/designs/gateway-triggers/wp/WP4-specs.md | 58 +++ .../designs/gateway-triggers/wp/WP4-status.md | 36 ++ docs/designs/gateway-triggers/wp/WP5-specs.md | 43 ++ .../designs/gateway-triggers/wp/WP5-status.md | 25 ++ docs/designs/gateway-triggers/wp/WP6-specs.md | 38 ++ .../designs/gateway-triggers/wp/WP6-status.md | 24 + 21 files changed, 2674 insertions(+) create mode 100644 docs/designs/gateway-triggers/gap.md create mode 100644 docs/designs/gateway-triggers/mapping.md create mode 100644 docs/designs/gateway-triggers/mimics.md create mode 100644 docs/designs/gateway-triggers/plan.md create mode 100644 docs/designs/gateway-triggers/proposal.md create mode 100644 docs/designs/gateway-triggers/research.md create mode 100644 docs/designs/gateway-triggers/wp/WL-runbook.md create mode 100644 docs/designs/gateway-triggers/wp/WP0-specs.md create mode 100644 docs/designs/gateway-triggers/wp/WP0-status.md create mode 100644 docs/designs/gateway-triggers/wp/WP1-specs.md create mode 100644 docs/designs/gateway-triggers/wp/WP1-status.md create mode 100644 docs/designs/gateway-triggers/wp/WP2-specs.md create mode 100644 docs/designs/gateway-triggers/wp/WP2-status.md create mode 100644 docs/designs/gateway-triggers/wp/WP3-specs.md create mode 100644 docs/designs/gateway-triggers/wp/WP3-status.md create mode 100644 docs/designs/gateway-triggers/wp/WP4-specs.md create mode 100644 docs/designs/gateway-triggers/wp/WP4-status.md create mode 100644 docs/designs/gateway-triggers/wp/WP5-specs.md create mode 100644 docs/designs/gateway-triggers/wp/WP5-status.md create mode 100644 docs/designs/gateway-triggers/wp/WP6-specs.md create mode 100644 docs/designs/gateway-triggers/wp/WP6-status.md diff --git a/docs/designs/gateway-triggers/gap.md b/docs/designs/gateway-triggers/gap.md new file mode 100644 index 0000000000..d3aca08511 --- /dev/null +++ b/docs/designs/gateway-triggers/gap.md @@ -0,0 +1,140 @@ +# Gateway Triggers — Gap + +The delta between **what exists today** and **what the proposal requires**. Every row is +something that must be built, moved, or decided; the "Source" column names what it is +patterned on (per `mimics.md`), and "Kind" classifies it: + +- **extract** — move shipped code into a shared home (the connection only). +- **mimic** — replicate an existing pattern in new triggers-domain files. +- **net-new** — no precedent; needs a design decision before code (per `mimics.md` § + Triggers vs Everything). +- **decision** — an open question to lock before or during build (from proposal § Risks + and `mapping.md` § Open questions). + +Nothing here changes the outbound `webhooks` domain or the `/tools` HTTP contract — both +are invariants (proposal § Success criteria). + +--- + +## 1. What exists today (the baseline) + +| Capability | Where | Reusable as-is? | +|---|---|---| +| Composio **auth** (initiate/status/refresh/revoke) | `ComposioToolsAdapter` (`core/tools/providers/composio/adapter.py`) | Yes — **extract** the auth verbs to the shared connection adapter | +| Connection persistence | `ToolConnectionDBE` / `tool_connections` (`dbs/postgres/tools/dbes.py:38`) | Yes — **rename** to `gateway_connections` (already domain-neutral) | +| Connection CRUD + OAuth callback | `ToolsService` (`core/tools/service.py:138-383`), `/tools/connections/...` + `/callback` (`router.py:785`) | Yes — **extract** to shared service; `/tools/connections` contract frozen | +| Action catalog (providers/integrations/actions) | `core/tools` catalog + `apis/fastapi/tools` | Pattern only — **mimic** for events | +| Composio call surface (httpx `_get/_post/_delete`, slug mapping) | `ComposioToolsAdapter` | Pattern only — **mimic** for the triggers REST surface | +| Two-table subscription/delivery model | `webhooks`: `webhook_subscriptions` + `webhook_deliveries` (`core/webhooks/`, `dbs/postgres/webhooks/`) | Pattern only — **mimic** (separate tables, no reuse) | +| DBA mixins for a subscription/delivery domain | `dbs/postgres/webhooks/dbas.py` | Pattern only — **mimic** (tools has no `dbas.py`) | +| Payload-mapping template + resolver | `payload_fields` + `resolve_payload_fields` (`core/webhooks/delivery.py:95`) → `resolve_json_selector` (`sdk/utils/resolvers.py:114`) | Resolver **reused** (promote + rename); template **mimicked** as `inputs_fields` | +| Inbound, signature-verified provider webhook | billing `POST /billing/stripe/events/` (`ee/.../billing/router.py:106,240`) | Pattern only — **mimic** the ingress shape | +| Workflow dispatch seam | `WorkflowsService.invoke_workflow` (`core/workflows/service.py:1698`) | Reused **as-is** — no new execution path | +| `env.composio` (api_key/api_url/enabled) | `utils/env.py:507`; wiring `entrypoints/routers.py:578` | Reused; **add** `COMPOSIO_WEBHOOK_SECRET` | + +> Tools never persisted a per-use record and webhooks never had a provider connection; +> **triggers is the first domain that needs both** a connection *and* a per-event standing +> record — which is why the connection is extracted (shared) and the subscription/delivery +> pair is mimicked (triggers-owned). + +--- + +## 2. The gap, by domain + +### 2.1 Shared `connections` domain (extract — A2-2) + +The connection moves out of `/tools` into a routerless shared domain. + +| # | Item | Kind | Source / note | +|---|---|---|---| +| C1 | `gateway_connections` table — rename `tool_connections` (+ `uq_`/`ix_`), no data transform | extract | `dbes.py:38`; table already domain-neutral | +| C2 | Migration authored **once in the shared `core_oss` chain** (runs in both editions), **not** the parked legacy `core` tree nor EE-only `core_ee` | extract | rename op only; `core` is frozen at `park00000000`; `gateway_connections` is shared schema. See `oss-ee-convergence/migration-chains-and-edition-switch.md` | +| C3 | `core/gateway/connections/` — service + DAO + interface, **no router** | extract | from `ToolsService` connection code (`service.py:138-383`) | +| C4 | `ConnectionsGatewayInterface` + Composio **auth** adapter (initiate/status/refresh/revoke) | extract | from `ComposioToolsAdapter` auth verbs | +| C5 | Repoint tools' connection auth at the shared service; `/tools/connections` contract frozen | extract | ~4 code refs: `dbes.py`, `dao.py:72`, `router.py:160` | +| C6 | `/tools/connections` and `/triggers/connections` both delegate to the one shared service over the same rows | mimic | no `/gateway/connections` route exists | +| C7 | **Cross-domain revoke rule**: revoke-for-everyone + show usage; deleting a subscription must not revoke the connection | net-new / decision | no prior connection had two consumers (`mimics.md` §6) | + +### 2.2 `triggers` domain — events catalog + adapter (mimic Tools) + +| # | Item | Kind | Source / note | +|---|---|---|---| +| E1 | Domain skeleton `apis/fastapi/triggers/`, `core/triggers/`, `dbs/postgres/triggers/` | mimic | tools layout | +| E2 | `ComposioTriggersAdapter` (own httpx client; `triggers_types`, `trigger_instances/...`) implementing `TriggersGatewayInterface` | mimic | `ComposioToolsAdapter` shape | +| E3 | Events catalog: `/triggers/catalog/.../integrations/{i}/events/{event_key}` returning the event's `trigger_config` schema | mimic | tools action catalog (`action → event`) | +| E4 | Wiring block in `entrypoints/routers.py` next to tools; adapter built only when `env.composio.enabled` | mimic | `routers.py:578` | +| E5 | **Exact Composio v3 REST paths** for trigger types/instances | decision | verify vs live OpenAPI (SDK names stable) | + +### 2.3 `triggers` domain — subscriptions + deliveries (mimic Webhooks) + +| # | Item | Kind | Source / note | +|---|---|---|---| +| S1 | `subscriptions` table: project-scoped, FlagsDBA (enabled/valid), DataDBA with `ti_*`, `trigger_config`, `inputs_fields`, destination `references`/`selector`, workflow ref; **FK → `gateway_connections`** | mimic | `webhook_subscriptions` (`types.py:116`) | +| S2 | `deliveries` table: one audit row per inbound event — resolved `inputs`, workflow `references`, `result`/`error`; migration defined once in `core_oss` | mimic | `webhook_deliveries` (`types.py:156`) | +| S3 | DBA mixins for both tables | mimic | `dbs/postgres/webhooks/dbas.py` (tools has none) | +| S4 | Subscription CRUD routes `/triggers/subscriptions/` · `/query` · `/{id}` · `/{id}/refresh` · `/{id}/revoke` + create/disable/delete the Composio `ti_*` via the adapter | mimic | `/webhooks/subscriptions/` + adapter calls | +| S5 | Delivery read routes `/triggers/deliveries` · `/{id}` · `/query` | mimic | `/webhooks/deliveries` | + +### 2.4 `triggers` domain — ingress (mimic Billing) + +| # | Item | Kind | Source / note | +|---|---|---|---| +| I1 | `POST /triggers/composio/events/` — read raw body before parsing | mimic | billing `/stripe/events/` | +| I2 | HMAC-SHA256 verify over `{id}.{ts}.{body}` with `COMPOSIO_WEBHOOK_SECRET`; 401 on bad sig; 200 no-op when secret unset | mimic | billing uses `stripe.Webhook.construct_event`; `research.md` § Webhook verification | +| I3 | Recover `project_id` from `metadata.user_id`; route `metadata.trigger_id` → local subscription; 200-skip unknown/disabled | mimic | billing's payload-scoping; `research.md` §1 | +| I4 | **Idempotency** dedup on `metadata.id` (store: column vs cache) | net-new / decision | billing leans on Stripe; we own it | +| I5 | Optional `target`-style env fan-out guard (one Composio webhook URL → many deployments) | decision | cf. `env.stripe.webhook_target` | +| I6 | **One-time project webhook-URL registration** with Composio (API vs dashboard, per-env) | net-new / decision | no precedent (`research.md` §4.2) | + +### 2.5 `triggers` domain — mapping + dispatch (mimic Webhooks resolver + net-new binding) + +| # | Item | Kind | Source / note | +|---|---|---|---| +| M1 | Promote `resolve_payload_fields` → `resolve_target_fields` into `agenta.sdk.utils.resolvers`; update the webhooks call site to the new name | mimic / extract | `mapping.md` §5/§6; lands at this point | +| M2 | `inputs_fields` template stored on the subscription; resolves into `WorkflowServiceRequest.data.inputs` **only** | mimic | `mapping.md` §3, §4.2 | +| M3 | `TRIGGER_EVENT_FIELDS` allowlist (event `data`/`type`/`timestamp`/curated `metadata`; never `ca_*`/secrets); context `{event, subscription, scope}` | mimic | `EVENT_CONTEXT_FIELDS` analogue | +| M4 | Destination = workflow `references` (+ `selector`), the `/retrieve` shape; drop into `request.references` at dispatch | mimic | `mapping.md` §4.1; `invoke_workflow` threads it (`service.py:556-557`) | +| M5 | **Trigger ↔ workflow binding** — store + resolve the workflow ref at dispatch | net-new | no domain binds a provider resource to a workflow | +| M6 | **System-initiated `invoke_workflow`** — what identity (`user_id`) a no-human invocation runs as | net-new / decision | seam only ever called request-scoped (`mimics.md` §2) | +| M7 | **Async dispatch** — ack-fast + enqueue vs inline (avoid webhook timeout → retry storm) | net-new / decision | proposal § Risks | +| M8 | **Default mapping** (`"$"` vs stricter) and **schema validation** of `inputs_fields` against the bound workflow's input schema | decision | `mapping.md` §6 | +| M9 | **Dispatch retry policy** for a failed invocation recorded in `deliveries` vs Composio redelivery | decision | `mapping.md` §6 | + +### 2.6 Frontend + +| # | Item | Kind | Source / note | +|---|---|---|---| +| F1 | "Triggers" surface on a connected integration: events browse, create subscription (pick event + bind workflow + mapping), list/disable/delete | mimic | tools UI (`web/.../gatewayTool`, `web/oss/.../settings/Tools`) | +| F2 | FE expects **overlapping connection reads** across `/tools/connections` and `/triggers/connections` (same rows) | net-new | consequence of A2-2 | +| F3 | Deliveries view (audit log) | mimic | could defer past v1 | + +--- + +## 3. Cross-cutting decisions to lock (consolidated) + +These appear above tagged `decision`; collected here because they gate multiple work items +and should be settled (some before code, some during). + +| Decision | Gates | Lean / default | Lock by | +|---|---|---|---| +| Exact Composio v3 REST paths (E5) | E2, E3, S4 | verify vs live OpenAPI | before adapter code | +| Project webhook-URL registration (I6) | ingress end-to-end test | manual setup step documented if API-less | before ingress test | +| Cross-domain revoke rule (C7) | C3–C6, F2 | revoke-for-everyone + show usage | before connection extract lands | +| Idempotency store (I4) | I-lane, dispatch | column on `deliveries` (dedup on `metadata.id`) | with deliveries table | +| Sync vs async dispatch (M7) | dispatch lane | async (ack-fast) | before dispatch code | +| System-initiated `user_id` (M6) | dispatch lane | a project-system identity (resolve from project) | before dispatch code | +| Default mapping + validation (M8) | subscription create | inputs-only default; validation = stretch | before subscription activate | +| Dispatch retry policy (M9) | deliveries semantics | bounded retries, else rely on Composio | with dispatch | + +--- + +## 4. Out of scope (restating non-goals so the gap isn't read as larger than it is) + +- No merge with / routing through the outbound `webhooks` domain. +- No workflow-hooks involvement. +- No downstream consumer beyond a single `invoke_workflow` per event (no eval/queue/re-emit). +- No new workflow execution path. +- No custom-OAuth ingress registration; managed-auth only. +- No polling fallback we own (Composio normalizes to one webhook). +- No SDK dependency (httpx direct, as tools). +- No EE-only gating beyond what tools already carry. diff --git a/docs/designs/gateway-triggers/mapping.md b/docs/designs/gateway-triggers/mapping.md new file mode 100644 index 0000000000..774508d77d --- /dev/null +++ b/docs/designs/gateway-triggers/mapping.md @@ -0,0 +1,330 @@ +# Gateway Triggers — Mapping & Config + +How the outbound **webhooks** domain lets a subscriber *shape the payload* it receives, +and how the same mechanism applies — in the opposite direction — to mapping an inbound +trigger **event** into a workflow invocation. + +This is the inbound dual of the webhook payload-mapping problem, so we copy the webhook +mechanism rather than invent one. + +--- + +## 1. How webhooks define their mapping today + +A webhook subscription stores a **payload template** and the delivery layer resolves it +against a curated **context** at send time. + +### The config field + +`WebhookSubscriptionData.payload_fields: Optional[Dict[str, Any]]` +(`core/webhooks/types.py:119`). It is an arbitrary JSON structure that doubles as a +template: leaves that are *selector strings* get replaced by values pulled from context; +everything else is passed through literally. + +### The context it resolves against + +At delivery, `prepare_webhook_request` (`core/webhooks/delivery.py:118`) builds a fixed, +**allowlisted** context: + +```python +context = { + "event": {k: v for k, v in event.items() if k in EVENT_CONTEXT_FIELDS}, + "subscription": {k: v for k, v in subscription.items() if k in SUBSCRIPTION_CONTEXT_FIELDS}, + "scope": {"project_id": str(project_id)}, +} +``` + +- `EVENT_CONTEXT_FIELDS` = `{event_id, event_type, timestamp, created_at, attributes}` +- `SUBSCRIPTION_CONTEXT_FIELDS` = `{id, name, tags, meta, created_at, updated_at}` + (`core/webhooks/types.py:26`) + +The allowlist is the security boundary: a subscriber's template can only reference these +keys, never arbitrary internal state. + +### The resolver (the template language) + +`resolve_payload_fields` (`delivery.py:95`) — to be renamed `resolve_target_fields` when +promoted to the SDK (§5/§6) — walks the template recursively; each leaf goes +through `resolve_json_selector` (`sdks/python/agenta/sdk/utils/resolvers.py:114`): + +- string starting with `$` → **JSONPath** against context +- string starting with `/` → **JSON Pointer** against context +- anything else (plain string, number, dict, list) → returned **as-is** (literal) +- resolution failure → `None` (never raises); depth-capped (`MAX_RESOLVE_DEPTH`) + +Default when `payload_fields is None`: `"$"` — i.e. deliver the whole context +(`delivery.py:149`). + +### Worked example (webhooks) + +Template stored on the subscription: + +```json +{ + "kind": "agenta.event", + "type": "$.event.event_type", + "when": "$.event.timestamp", + "project": "$.scope.project_id", + "sub": "$.subscription.name" +} +``` + +Resolved and POSTed to the subscriber URL: + +```json +{ + "kind": "agenta.event", + "type": "traces.queried", + "when": "2026-06-18T10:00:00Z", + "project": "019abc...", + "sub": "my-prod-hook" +} +``` + +So the webhook "mapping" is: **subscriber-authored JSON template + selectors over an +allowlisted context, resolved at delivery.** Static where the subscriber wants constants, +dynamic where they reference `$.event.*` / `$.subscription.*` / `$.scope.*`. + +--- + +## 2. Decompose the webhook subscription: three independent concerns + +`WebhookSubscriptionData` (`core/webhooks/types.py:116`) bundles three concerns that are +actually independent. Separating them is the key to seeing what carries over to triggers +unchanged and what genuinely differs: + +```python +class WebhookSubscriptionData(BaseModel): + url, headers, auth_mode # DESTINATION — where/how to deliver + payload_fields # MAPPING — how to shape the body + event_types # FILTER — which events +``` + +| Concern | Webhook field | Carries to triggers? | +|---------|---------------|----------------------| +| **filter** — which events | `event_types` | **same idea** — which provider event this subscription watches | +| **mapping** — shape the data | `payload_fields` | **same mechanism** — identical resolver + context; the field is named `inputs_fields` because it maps into `data.inputs`, not a whole body (§3, §4.2) | +| **destination** — where it goes | `url`, `headers`, `auth_mode` | **different** — a workflow `references` + `selector`, not a by-value URL (§4.1) | + +So the answer to "why would mapping/context differ?": the **mechanism and context don't** +(same resolver, same `{event, subscription, scope}`). Two things do, and both follow from +the target being an internal workflow rather than an external URL: the **destination** is a +`references`/`selector` (§4.1), and the mapping field maps into **`data.inputs`** rather +than a whole HTTP body, so it is named `inputs_fields` (§4.2). + +--- + +## 3. Same mapping mechanism + context; field named for its target + +### The field — `inputs_fields` (webhooks' `payload_fields`, retargeted) + +Triggers store the **same kind of template** webhooks store in `payload_fields`: a JSON +structure with `$`/`/` selectors over context, same resolver, same default. The **field is +named `inputs_fields`** rather than `payload_fields` because it maps into +`WorkflowServiceRequest.data.inputs` (§4.2), not a whole HTTP body. The name states the +target — the same reason webhooks' field is called *payload*_fields (it maps the payload). + +```text +webhooks subscription: payload_fields → whole HTTP body +triggers subscription: inputs_fields → request.data.inputs +``` + +Mechanism, resolver, and context are identical; only the field name and its target differ. + +### Same context — `{event, subscription, scope}` + +Resist the temptation to expose the raw Composio envelope (`{data, metadata}`) directly. +Keep the **identical three-slot, allowlisted** context webhooks uses — the slots just bind +to the inbound analogues: + +| Slot | Webhooks (outbound) | Triggers (inbound) | +|------|---------------------|--------------------| +| `event` | the Agenta event that fired (allowlisted) | the verified provider event that arrived (allowlisted) | +| `subscription` | the webhook subscription (allowlisted) | the trigger subscription (allowlisted) | +| `scope` | `{project_id}` | `{project_id}` (recovered from `metadata.user_id`) | + +```python +# triggers — same shape as webhooks' prepare_webhook_request context +context = { + "event": {k: v for k, v in inbound_event.items() if k in TRIGGER_EVENT_FIELDS}, + "subscription": {k: v for k, v in subscription.items() if k in SUBSCRIPTION_CONTEXT_FIELDS}, + "scope": {"project_id": str(project_id)}, +} +``` + +`TRIGGER_EVENT_FIELDS` is the triggers analogue of `EVENT_CONTEXT_FIELDS` — an allowlist +over the inbound event (its `data`, `type`, `timestamp`, and curated `metadata` like +`trigger_slug`/`trigger_id`/`toolkit_slug`), never exposing `ca_*`, secrets, or connection +internals. Same discipline, same security boundary, identical resolver +(`resolve_target_fields` → `resolve_json_selector`, `$`/`/` selectors, literal +passthrough, null-on-miss). + +### Worked example (triggers) + +Subscription `inputs_fields` (Gmail "new message" → a triage workflow): + +```json +{ + "subject": "$.event.data.subject", + "from": "$.event.data.from", + "body": "$.event.data.message_text", + "received": "$.event.timestamp", + "watch": "$.subscription.name", + "source": "gmail" +} +``` + +Inbound event at `/triggers/composio/events/` (its allowlisted form becomes `context.event`), +resolved to: + +```json +{ + "subject": "Refund?", "from": "a@x.com", "body": "...", + "received": "2026-06-18T10:00:00Z", "watch": "support-triage", "source": "gmail" +} +``` + +**Important — this resolved object is *not* the whole request.** It becomes only +`WorkflowServiceRequest.data.inputs` (§4.2). The destination (which workflow) comes from a +separately-stored reference (§4.1), and the envelope/auth is filled by `invoke_workflow`. + +--- + +## 4. The two real differences: destination, and *what* the payload maps into + +The actual `invoke_workflow` request type is `WorkflowServiceRequest` +(= `WorkflowInvokeRequest`, `sdks/python/agenta/sdk/models/workflows.py:257-262`): + +```python +WorkflowBaseRequest: + version + references: Dict[str, Reference] # WHICH workflow/revision ← destination + links: Dict[str, Link] + selector: Selector # which slice to extract + secrets, credentials # auth — filled by invoke_workflow internally +WorkflowInvokeRequest(WorkflowBaseRequest): + data: WorkflowRequestData + revision, parameters, testcase, inputs, trace, outputs # the payload area +``` + +This makes two things precise that a naive "webhooks but inbound" framing gets wrong. + +### 4.1 Destination = `references` (+ `selector`), the existing /retrieve shape + +A webhook's destination is described **by value** — `url`, `headers`, `auth_mode` inline. +A trigger's destination is an Agenta **workflow**, an internal entity, so it is described +**by reference** using the **same `Reference` / `Selector` primitives the `/retrieve` and +inspect paths already use** — not an ad-hoc `{workflow_id, ...}`. + +`Reference(Identifier, Slug, Version)` = `{ id?, slug?, version? }` +(`sdks/.../models/shared.py:102`). `invoke_workflow` already threads +`request.references` / `request.selector` straight through (`service.py:556-557`). + +So the subscription stores a workflow **reference** (+ optional selector), and dispatch +drops it into `request.references`: + +```text +webhook destination: { url, headers, auth_mode } ← by value +trigger destination: references: { "workflow": Reference{id|slug, version} } [+ selector] + ← by reference, same as /retrieve +``` + +No new addressing scheme — reuse how workflows are referenced everywhere else. + +### 4.2 The mapping (`inputs_fields`) maps into `data.inputs`, NOT the whole request + +For **webhooks**, `payload_fields` maps to the **entire** HTTP body — an HTTP POST body +*is* the payload; there is nothing else. + +For **triggers**, the request envelope has dedicated structural slots — `references` +(destination, §4.1), `version`, `secrets`/`credentials` (auth, internal). The mapping must +**not** produce those. It produces only the "data fed in" slot, hence the field name +`inputs_fields`: + +```text +WorkflowServiceRequest +├─ references / selector ← destination (from §4.1; NOT from inputs_fields) +├─ version, secrets, credentials ← envelope/auth (internal; NOT mapped) +└─ data: WorkflowRequestData + └─ inputs ◄──────────────── inputs_fields resolves into HERE (and only here) +``` + +So the asymmetry, stated exactly: + +```text +webhooks: payload_fields → the whole HTTP body +triggers: inputs_fields → request.data.inputs (a sub-field of the request) +``` + +Whether any *other* `data.*` sub-fields are mappable (`parameters`? `testcase`?) is an open +call (§6); the safe default is **inputs only**. + +### 4.3 Deliveries (same pair, different fields) + +Webhooks is a **two-table** domain: `webhook_subscriptions` (the standing config) **and** +`webhook_deliveries` (one audit row per attempt) — `WebhookDelivery` / +`WebhookDeliveryData{url, headers, payload, response, error}` (`types.py:156`), with routes +`/webhooks/deliveries`, `/{id}`, `/query` (`router.py:110`). + +Triggers mirrors the pair: `subscriptions` **and** `deliveries`. A delivery row records one +inbound event being dispatched to its workflow — the by-reference destination, the resolved +inputs, and the outcome: + +```text +WebhookDeliveryData { url, headers, payload, response{status_code, body}, error } +TriggerDeliveryData { references (workflow), inputs (resolved inputs_fields), result, error } +``` + +This is the right call (not "maybe"): a delivery record is needed precisely for the cases +where the workflow's own trace does **not** exist — dispatch that fails *before* invocation +(bad mapping, workflow not found, connection invalid) or is deduped/skipped. It is also the +retry and observability surface, exactly as `webhook_deliveries` is for the outbound side. +Full table/route symmetry in `mimics.md` § Triggers vs Webhooks. + +--- + +## 5. What we reuse vs. what's new + +| Piece | Status | +|-------|--------| +| Mapping field | **same mechanism, retargeted name** — `inputs_fields` (vs. `payload_fields`); maps `data.inputs`, not a whole body | +| Context shape `{event, subscription, scope}` + allowlist discipline | **identical** — define `TRIGGER_EVENT_FIELDS` like `EVENT_CONTEXT_FIELDS`; reuse `SUBSCRIPTION_CONTEXT_FIELDS` | +| Selector resolver (`resolve_json_selector`) | **reuse** — already in `agenta.sdk.utils.resolvers` | +| Recursive template walk (`resolve_payload_fields` → `resolve_target_fields`) | **reuse + rename** — promote from `core/webhooks/delivery.py` to the SDK under the neutral name `resolve_target_fields`, so both domains consume it (avoids triggers→webhooks import) | +| `event_types` filter | **same idea** — which provider event the subscription watches | +| Destination | **reuse a different primitive** — workflow `Reference`/`Selector` (the `/retrieve` shape) instead of `url/headers/auth_mode` | +| Mapping *target* | **different** — `inputs_fields` resolves into `data.inputs` only, not the whole request (webhooks maps the whole body) | +| Two-table domain (subscriptions + deliveries) | **same shape** — `subscriptions` + `deliveries`, mirroring `webhook_subscriptions` + `webhook_deliveries` | +| Delivery record fields | **different fields, same idea** — `references + inputs + result` vs. `url + payload + response` | + +Net: **the resolver, the mapping mechanism, and the `{event, subscription, scope}` +context are reused/identical**, and like webhooks it is a **two-table** domain +(subscriptions + deliveries). The real differences all follow from the target being an +internal workflow: (a) the destination is a workflow *reference* (the `/retrieve` +`Reference`/`Selector`, not a by-value URL), and (b) the mapping field is `inputs_fields` +landing in `data.inputs`, not the whole body. + +--- + +## 6. Open questions + +- **Default mapping** — webhooks defaults `payload_fields` to `"$"` (whole context). + Triggers feeding a *typed* workflow may want a stricter `inputs_fields` default (e.g. + `"$.event.data"`) or require an explicit mapping before the subscription can activate. +- **Validation against the workflow's input schema** — should creating a subscription + validate `inputs_fields`' resolved shape against the bound workflow revision's expected + inputs? Webhooks has no downstream schema to check; triggers does — a new opportunity and + a new failure mode. +- **Delivery retries** — webhooks has `WEBHOOK_MAX_RETRIES = 5` on the outbound leg. What + is the retry policy for a failed *dispatch* (workflow invocation) recorded in + `deliveries`, vs. relying on Composio's own inbound redelivery? (The `deliveries` table + itself is decided — see §4.3.) +- **`TRIGGER_EVENT_FIELDS` contents** — which inbound-event keys to expose + (`data`, `type`, `timestamp`, curated `metadata`); keep `ca_*`/secrets out. +- **Resolver location + rename** — `resolve_payload_fields` lives in the webhooks domain; + promote it next to `resolve_json_selector` in `agenta.sdk.utils.resolvers` under the + neutral name **`resolve_target_fields`** (it resolves a template into *a* target, + whichever consumer's — whole body for webhooks, `data.inputs` for triggers), so triggers + and webhooks both consume it from the SDK. The webhooks call site updates to the new name + at that point — a docs-level decision now; the actual rename lands when the SDK promotion + happens (during the triggers build). diff --git a/docs/designs/gateway-triggers/mimics.md b/docs/designs/gateway-triggers/mimics.md new file mode 100644 index 0000000000..74e6a4c0b7 --- /dev/null +++ b/docs/designs/gateway-triggers/mimics.md @@ -0,0 +1,307 @@ +# Gateway Triggers — Mimics & Contrasts + +This doc maps each part of the work onto the existing Agenta pattern it relates to. +Two relationship kinds are used, and they are different: + +- **mimic** — *replicate the pattern in new triggers-domain files* (copy structure, swap + nouns; no imports across the boundary). Applies to events catalog, subscriptions, + ingress, dispatch. +- **share/extract** — *the same code/table serves both domains.* Applies to **one** thing + only: provider **connections** (`ca_*`), which are pulled out of `/tools` into a shared + `connections` domain and consumed by both (decision **A2-2**). + +Terminology: the triggers catalog leaf is an **event** (≈ a tools **action**). The created +state is **two** records with **different owners**: + +- **connection** — durable provider auth (`ca_*`). A **shared, gateway-level** record + (`gateway_connections`, renamed from `tool_connections`), used by both tools and + triggers. Not triggers-owned. +- **subscription** — a standing watch on one event (`ti_*` + config + workflow, FK → + connection), owned by the triggers domain. Modeled on a webhook subscription. Split from + the connection because one `ca_*` backs many `ti_*`. + +This file is organized as a set of pairwise comparisons: + +- [Triggers vs Tools](#triggers-vs-tools) — the structural template (events catalog, adapter) + the **shared** connection (extracted from tools) +- [Triggers vs Billing](#triggers-vs-billing) — the inbound-event ingress template +- [Triggers vs Webhooks](#triggers-vs-webhooks) — the two **subscription** species + the directional mirror +- [Triggers vs Everything (the net-new parts)](#triggers-vs-everything-the-net-new-parts) + +A one-line map of where each part comes from: + +| Part | Relationship | Source | +|------|--------------|--------| +| **event** catalog, triggers adapter, domain layout | mimic | **Tools** | +| provider **connection** (`ca_*`) | **share/extract** | **Tools** → shared `gateway_connections` | +| the **subscription** + **delivery** tables (two-table domain, CRUD, lifecycle) | mimic | **Webhooks** (`webhook_subscriptions` + `webhook_deliveries`) | +| inbound event endpoint, signature verify, payload-based scoping | mimic | **Billing** (Stripe `/stripe/events/`) | +| trigger↔workflow binding, system-initiated dispatch, idempotency | net new | **nothing** | + +> **Two parents, plus one shared organ.** The triggers code is a cross of **tools** +> (catalog/adapter machinery) and **webhooks** (the subscription model + lifecycle); the +> ingress endpoint comes from **billing**. Separately, the provider **connection** is not +> re-created at all — it is extracted from tools into a shared `connections` domain that +> both tools and triggers sit on (A2-2). The one sanctioned cross-domain runtime calls are +> triggers → the shared connections service (auth) and triggers → +> `WorkflowsService.invoke_workflow` (dispatch). + +--- + +## Triggers vs Tools + +Tools relates to triggers in **two** different ways, and it's important not to conflate +them: + +- **mimic** — the triggers *event catalog* and *Composio adapter* replicate the tools + catalog/adapter structure in new files. +- **share/extract** — the tools *connection* is not copied; it is **moved** into a shared + `connections` domain that both tools and triggers consume. + +### Part A — mimic: events catalog + triggers adapter + +New triggers-domain files, modeled on tools, swapping `action → event`: + +| Aspect | `/tools` | `/triggers` (new files, same shape) | +|--------|----------|-------------------------------------| +| Domain layout | `apis/fastapi/tools/`, `core/tools/`, `dbs/postgres/tools/` | `apis/fastapi/triggers/`, `core/triggers/`, `dbs/postgres/triggers/` | +| Layering | Router → Service → DAOInterface + GatewayInterface → impls | identical | +| Wiring | `tools` block in `entrypoints/routers.py:578` | `triggers` block next to it | +| Adapter | `ComposioToolsAdapter` (httpx, no SDK) | own `ComposioTriggersAdapter` (httpx, no SDK) | +| Catalog leaf | **actions** + `input_parameters` schema | **events** + `trigger_config` schema | +| Catalog route | `.../integrations/{i}/actions/{action_key}` | `.../integrations/{i}/events/{event_key}` | +| Env gate | `env.composio` | `env.composio` (shared value) + `COMPOSIO_WEBHOOK_SECRET` | + +### Part B — share/extract: the provider connection + +The tools connection (`ca_*`, OAuth, refresh, revoke) is **the same object** triggers +needs for auth. Rather than re-create it, extract it from `/tools` into a shared +`connections` domain (decision A2-2): + +| Aspect | before (tools-owned) | after (shared) | +|--------|----------------------|----------------| +| Table | `tool_connections` | `gateway_connections` (renamed; already domain-neutral) | +| Code | `core/tools` connection code + `ComposioToolsAdapter` auth methods | `core/gateway/connections/` + a `ConnectionsGatewayInterface` auth adapter | +| Router | `/tools/connections` router | **none of its own** — shared service has no router | +| HTTP surface | `/tools/connections` | `/tools/connections` **and** `/triggers/connections`, both delegating to the shared service (same rows) | +| Auth verbs | `initiate_connection`, `refresh`, `revoke`, `get_status` | unchanged, now in the shared service | +| Consumers | tools only | tools **and** triggers | + +The tools `/tools/connections` HTTP contract is unchanged; its handlers delegate to the +shared service. `ToolsService` connection management (`core/tools/service.py:138-383`) is +the code that *moves* (lightly generalized), not code that triggers re-creates. + +### Where they differ + +| | Tools | Triggers | +|---|-------|----------| +| Direction | outbound (we call the provider) | inbound (the provider calls us) | +| Source of work | an LLM/agent tool call | a provider event | +| Per-event work | synchronous response to caller | invoke the bound Agenta workflow | +| Per-use record | *(ephemeral tool call — nothing persisted)* | a **subscription** (`ti_*` + config + workflow), FK → shared connection | +| Relation to connection | uses it directly to call actions | references it from a standing subscription | +| Extra surface | — | an inbound ingress endpoint (no tools analogue — see Billing) | + +> **Connect once, used by both.** Because the connection is shared, a Gmail connected for +> tools is immediately usable by triggers and vice-versa — no second OAuth consent. The +> cost is a cross-domain revoke rule (revoking `ca_*` affects both; deleting a subscription +> must not revoke the connection). This is the inverse of rejected option B, where each +> domain owned its own connection and the user connected twice (see +> [Triggers vs Everything](#triggers-vs-everything-the-net-new-parts) and +> `proposal.md` § Alternatives). + +--- + +## Triggers vs Billing + +**Relationship: the ingress template.** The inbound event endpoint has **no analogue in +tools** (tools are outbound). Its only precedent in the codebase is billing's Stripe +webhook — Agenta's one existing inbound, signature-verified provider-event handler. This +is the most important pattern to copy correctly. + +Reference: `handle_events` at `api/ee/src/apis/fastapi/billing/router.py:240`, route at +`:106`. + +### What lines up (billing) + +| Aspect | `/billing` (Stripe) | `/triggers` (Composio) | +|--------|---------------------|------------------------| +| Route shape | `POST /billing/stripe/events/` | `POST /triggers/composio/events/` | +| Convention | `{domain}/{provider}/events/` | same | +| Body handling | `await request.body()` before parsing | same — raw body required for verify | +| Verification | `stripe.Webhook.construct_event(payload, sig, env.stripe.webhook_secret)` | HMAC-SHA256 over `{id}.{ts}.{body}`, `COMPOSIO_WEBHOOK_SECRET` | +| Bad signature | 401, return | 401, return | +| Unconfigured provider | 200 no-op (`"Stripe not configured"`) | 200 no-op if secret unset | +| Irrelevant/skipped event | 200 skip (so provider stops retrying) | 200 skip (unknown `trigger_id`, disabled, duplicate) | +| Tenant scope | from payload `metadata.organization_id` | from payload `metadata.user_id` → `project_id` | +| Routing key | event `type` | `metadata.trigger_id` → local row | +| Env fan-out guard | `metadata.target == env.stripe.webhook_target` | optional `target`-style guard (see below) | +| Boundary decorator | `@intercept_exceptions()` | same | + +Handler skeleton to lift: + +```python +payload = await request.body() # raw body BEFORE parsing — required for verify +# verify provider signature against raw body + secret; on failure → 401 + return +# extract scope from the payload, look up the local record, act +# always 2xx for events you intentionally skip (so the provider doesn't retry) +``` + +### Where they differ (billing) + +| | Billing (Stripe) | Triggers (Composio) | +|---|------------------|---------------------| +| Scope key | `organization_id` | `project_id` (from `user_id`) | +| What the event drives | subscription/meter state changes | invoke an Agenta workflow | +| Processing | effectively synchronous in-handler | likely ack-fast + async dispatch (avoid webhook timeout/retry storms) | +| Dedup | relies on Stripe semantics | **we** dedup on `metadata.id` (new) | +| Edition | EE-only | wherever tools ship | + +> **Worth copying: the `webhook_target` filter.** Stripe lets one account fan out to +> dev/staging/prod without cross-talk by checking `metadata.target` against +> `env.stripe.webhook_target`. One Composio project's single webhook URL serving multiple +> Agenta deployments has the same need — a `target`-style guard is a reasonable copy. + +--- + +## Triggers vs Webhooks + +**Relationship: the subscription + delivery model — and the conceptual mirror.** The +outbound `webhooks` domain (`api/oss/src/core/webhooks/`) matters to triggers in two +distinct ways: it owns the **two-table subscription/delivery** model the trigger records +are patterned on, *and* it is the directional mirror of the whole feature. As always: copy +the pattern into new files, do not touch `core/webhooks/`. + +Webhooks is a **two-table domain**: `webhook_subscriptions` (standing config) + +`webhook_deliveries` (one audit row per attempt). Triggers mirrors the **same pair**: +`subscriptions` + `deliveries`. + +### Part A1 — the two subscription species + +A **webhook subscription** already exists: a project subscribes to internal Agenta events +and they are delivered *out* to a URL. A **trigger subscription** is the inbound dual: a +project subscribes to provider events and they are delivered *in* to a workflow. Same +noun, same lifecycle shape, opposite direction. + +Webhook subscription shape — `WebhookSubscription` / +`WebhookSubscriptionData{url, event_types, auth_mode, secret, payload_fields}` (`core/webhooks/types.py:116`), +routes `/webhooks/subscriptions/` · `/query` · `/{id}` · `/{id}/test` +(`apis/fastapi/webhooks/router.py:55`). + +| Aspect | webhook subscription | trigger subscription | +|--------|----------------------|----------------------| +| Noun / table | `webhook_subscriptions` | `subscriptions` (triggers domain) | +| Routes | `/webhooks/subscriptions/` + `/query` + `/{id}` + `/{id}/test` | `/triggers/subscriptions/` + `/query` + `/{id}` + `/{id}/refresh` + `/{id}/revoke` | +| What you subscribe to | internal `EventType`s (`event_types`) | a provider **event** (Composio trigger type) | +| Direction | event delivered **out** to `data.url` | event delivered **in**, dispatched to a workflow | +| Destination | customer URL (`url/headers/auth_mode`, by value) | workflow `references` + `selector` (by reference) | +| Mapping field | `payload_fields` → whole body | `inputs_fields` → `data.inputs` (see `mapping.md`) | +| Secret | `secret` / `secret_id` (we sign outgoing) | `COMPOSIO_WEBHOOK_SECRET` (we verify incoming) | +| Project-scoped record w/ lifecycle | yes | yes | +| Mixins | `Identifier, Lifecycle, Header, Metadata` | same + `FlagsDBA`, `DataDBA` for `ti_*` + config + workflow ref + FK → connection | + +### Part A2 — the two delivery species + +`webhook_deliveries` records each outbound attempt; `deliveries` (triggers) records each +inbound event dispatched to its workflow. Same role (audit + retry surface), fields differ +only where the destination differs. + +`WebhookDelivery` / `WebhookDeliveryData{url, headers, payload, response{status_code, body}, error}` +(`core/webhooks/types.py:156`), routes `/webhooks/deliveries` · `/{id}` · `/query` +(`router.py:110`). + +| Aspect | webhook delivery | trigger delivery | +|--------|------------------|------------------| +| Table | `webhook_deliveries` | `deliveries` (triggers domain) | +| Routes | `/webhooks/deliveries` · `/{id}` · `/query` | `/triggers/deliveries` · `/{id}` · `/query` | +| One row per | outbound POST attempt | inbound event dispatched | +| Destination fields | `url`, `headers` | `references` (workflow) | +| Payload fields | `payload` (sent body) | `inputs` (resolved `inputs_fields`) | +| Outcome fields | `response{status_code, body}`, `error` | `result`, `error` | +| Why it exists | audit + retry of a failed POST | audit + retry of a failed dispatch — and the **only** record when dispatch fails *before* invocation (bad mapping, workflow not found), where no workflow trace exists | + +> The trigger `deliveries` table is **decided, not optional** — it is the dual of +> `webhook_deliveries`, and it is the sole audit/retry surface for dispatches that never +> reach the workflow. (Reasoning in `mapping.md` §4.3.) + +A trigger subscription is modeled on a webhook subscription for its **subscribe-to-events +lifecycle** (a project-scoped record naming what to watch, with CRUD + a secret). It does +**not** carry the provider auth — that lives in the shared `gateway_connections` row it +FKs to (A2-2). So: + +```text +trigger subscription = webhook subscription (subscribe to an event, /subscriptions CRUD, lifecycle) + + FK → shared connection (provider auth: ca_*, in the connections domain) + + workflow binding (net-new — see last section) +``` + +The connection half is **shared, not bundled** — see [Triggers vs Tools, Part B](#part-b--shareextract-the-provider-connection). + +### Part B — the directional mirror (the framing) + +```text +outbound webhooks: Agenta event ──▶ customer URL (we sign + POST out) +gateway triggers: provider event ──▶ Agenta workflow (we verify + invoke in) +``` + +As `webhooks` is to Agenta events, triggers are to provider events — pointed inward and +ending in a workflow. + +| | Outbound `webhooks` | Triggers | +|---|---------------------|----------| +| Direction | sender (Agenta → customer) | receiver (Composio → Agenta) | +| HMAC role | we **sign** outgoing | we **verify** incoming | +| Where the "subscription" lives | the Agenta `webhook_subscriptions` row | the Agenta `subscriptions` row **and** a Composio trigger instance it mirrors | +| Deliveries/retries | owned here (`WEBHOOK_MAX_RETRIES = 5`, delivery records) | inbound leg owned by Composio; our dispatch is the new part | +| Destination | an arbitrary customer URL | an Agenta workflow | +| Event source | internal `EventType`s | external provider events | +| Code reuse | **none** — must not route through it | — | + +> Despite the shared "subscription" noun and lifecycle, do **not** route trigger ingress +> through the webhooks subscription/delivery machinery, and do not share its tables. They +> are separate domains that happen to be duals — the similarity is a pattern to copy, not +> code to reuse. + +--- + +## Triggers vs Everything (the net-new parts) + +These have **no precedent** in tools, billing, or webhooks. They must be designed, and +they deserve the most review. + +1. **Trigger ↔ workflow binding.** Storing a workflow ref (workflow + + revision/environment) on the trigger row and resolving it at dispatch. Nothing in any + domain binds a provider resource to a workflow. + +2. **System-initiated `invoke_workflow`.** The seam exists + (`WorkflowsService.invoke_workflow`, `core/workflows/service.py:1698`) but has only + been called from human-initiated, request-scoped paths. A no-human, event-triggered + invocation is new — what identity it runs as is an open decision (proposal §Risks). + +3. **Event → `WorkflowServiceRequest` mapping.** Shaping an arbitrary provider event + payload into workflow inputs. No existing code maps external JSON into a workflow + request; the schema-mapping question is non-trivial. + +4. **Async dispatch + idempotency.** Billing's handler is effectively synchronous and + leans on Stripe's dedup. Invoking a workflow inline risks webhook timeouts → provider + retries → duplicate runs. Ack-fast-then-dispatch + `metadata.id` dedup is new behavior. + +5. **One-time project webhook-URL registration with Composio.** Tools never registered an + *inbound* URL with a provider; Stripe's is configured out-of-band in its dashboard. + How Composio's is registered (API vs dashboard) and managed per-environment is new + operational surface. + +6. **Connection extraction + cross-domain revoke (A2-2).** Pulling `tool_connections` out + into a shared `gateway_connections` domain is a migration + repoint of shipped tools + code (cheap — the table is already domain-neutral, ~4 refs). The genuinely *new + behavior* is the cross-domain lifecycle rule: revoking a shared `ca_*` affects both + tools and triggers (lean: revoke-for-everyone + show usage), and deleting a subscription + must not revoke the connection. No prior domain had a connection with two consumers. + +> Rule of thumb by relationship kind: +> - **mimic** (Tools §A events/adapter, Webhooks subscription, Billing ingress) — replicate +> the named file's structure into a new triggers-domain file and adjust nouns; never +> import or subclass across the boundary. +> - **share/extract** (Tools §B connection) — move the code into the shared `connections` +> domain and have both tools and triggers depend on it; the shared service *is* imported +> by both (that's the point). +> - **net new** (this section) — needs a design decision before code. diff --git a/docs/designs/gateway-triggers/plan.md b/docs/designs/gateway-triggers/plan.md new file mode 100644 index 0000000000..74a6f26830 --- /dev/null +++ b/docs/designs/gateway-triggers/plan.md @@ -0,0 +1,409 @@ +# Gateway Triggers — Plan + +Work breakdown for the gap (`gap.md`). The work splits into seven units; we look at them +through **three different lenses**, each with its own dependency semantics. Same seven units +in every view — only the edges differ. + +| View | Unit | Edge means | Fan-in? | Answers | +|------|------|-----------|---------|---------| +| **Work Packages** (WP) | a unit of functionality | *X functionally needs Y* (code/data dependency) | **yes** — the true DAG | what depends on what | +| **Work Lanes** (WL) | a GitButler branch | *X is `--anchor`ed on Y* (merge/review tree) | **no** — one parent per branch | how it merges | +| **Work Streams** (WS) | a parallel build assignment | *X builds against Y's frozen contract* (stub until merged) | n/a — all run at once | who builds what concurrently | + +Each WP closes a set of `gap.md` items and is independently **reviewable** (a coherent diff) +and **functional** (does something real and testable on its own) — see §3 for per-package +detail. The same unit carries one id in each view: a package is `WP{k}`, its lane node +`WL{k}`, its stream slot `WS{k}` (same `k`). + +**The seven units** (full scope in §3): + +| k | Unit | Area | +|---|------|------| +| 0 | Connection extract (A2-2): shared `gateway_connections` + service | api (touches shipped tools) | +| 1 | Events catalog + `ComposioTriggersAdapter` | api | +| 2 | Resolver promotion to SDK (`resolve_target_fields`) | sdk + webhooks | +| 3 | Subscriptions + deliveries tables + CRUD | api | +| 4 | Ingress + dispatch (receive → resolve → invoke → record) | api | +| 5 | Web: catalog + connections UI | web | +| 6 | Web: subscriptions + deliveries UI | web | + +--- + +## 1. Work Packages — functional dependencies (the true DAG, fan-in allowed) + +What each unit needs to *work*, from the data model and call graph. This is the ground truth; +the other two views are derived from it. Fan-in is real here — a node can need two others. + +```text +WP0 ─────────────┬──────────────▶ WP3 ──────────┬──────────▶ WP4 +(gateway_conns) │ (FK + adapter) ▲ │ (tables) + │ │ │ ▲ +WP1 ──┬──────────┘ │ │ │ +(catalog+adapter) │ (adapter)──────┘ │ │ + │ └────────────────────────▶ WP5 │ │ + │ (catalog+conns) │ │ +WP2 ──────────────────────────────────────────────┘ │ (resolver) +(resolver→SDK) │ + WP6 ─┘ + (subs/deliveries API ← WP3) +``` + +Edges (X ← Y reads "X functionally needs Y"): + +- **WP3 ← WP0** — `subscriptions` FKs `gateway_connections` (gap S1). +- **WP3 ← WP1** — creating the `ti_*` calls `ComposioTriggersAdapter.create_subscription` + (the *adapter*, not the catalog routes). → WP3 fans in on {WP0, WP1}. +- **WP4 ← WP3** — dispatch reads a subscription, writes a delivery row. +- **WP4 ← WP2** — dispatch imports the promoted `resolve_target_fields`. → WP4 fans in on + {WP3, WP2}. +- **WP5 ← WP1** (catalog API) **and ← WP0** (the `/…/connections` view over + `gateway_connections`). → WP5 fans in on {WP1, WP0}. +- **WP6 ← WP3** — the `/triggers/subscriptions` + `/triggers/deliveries` API. +- **WP0, WP1, WP2** — no in-feature dependency (roots). + +--- + +## 2. Work Lanes — merge tree (GitButler `--anchor`, no fan-in) + +A GitButler series is linear: each branch has exactly **one** `--anchor` parent (two parents +is a merge commit, which collapses the stack — `vibes/AGENTS.md`: "series need linear +history"). So the WP DAG must be **projected onto a tree**: every WP fan-in is resolved by +anchoring on *one* functional parent; the other functional parent(s) must simply be a +**transitive ancestor** in the tree (so the needed code is present in the branch). Fan-**out** +is allowed (a parent may have many children). + +The constraint that shapes the tree: **WP4 needs WP2's resolver**, so WP2 must sit on the +line *below* WP4 (an ancestor), not on a sibling branch — otherwise that edge would be a +fan-in the tree can't hold. Placing WP2 between WP1 and WP3 satisfies it: + +```text +main +└─ WL0 wp0-connections-extract + └─ WL1 wp1-events-catalog --anchor wp0 + ├─ WL2 wp2-resolver-promote --anchor wp1 (on the WL4 line, so WP2 is WL4's ancestor) + │ └─ WL3 wp3-subscriptions --anchor wp2 (ancestors wp2,wp1,wp0 ✓ cover WP0+WP1) + │ ├─ WL4 wp4-ingress-dispatch --anchor wp3 (ancestors incl. wp2 ✓ + wp3 ✓) + │ └─ WL6 wp6-web-subscriptions --anchor wp3 + └─ WL5 wp5-web-catalog --anchor wp1 (ancestors wp1,wp0 ✓) +``` + +**Every functional edge from §1 is covered by a tree ancestor**, with no branch having two +parents: + +| WP needs | satisfied in tree by | +|----------|----------------------| +| WP3 ← WP0, WP1 | WL3 anchored on WL2; WL0, WL1 are ancestors | +| WP4 ← WP3, WP2 | WL4 anchored on WL3; WL2 is an ancestor | +| WP5 ← WP1, WP0 | WL5 anchored on WL1; WL0 is an ancestor | +| WP6 ← WP3 | WL6 anchored on WL3 | + +Each PR sets `--base` to its anchor so the diff stays scoped. Merge is bottom-up along the +tree; because every dependency is a structural ancestor, **no cross-branch merge-order +coordination is required** — the property we couldn't get from parallel lanes. + +> Trade-off of the tree: it linearizes WP2 and WP5/WP6 under the WP1 line. That is a *merge* +> topology only — it does **not** mean they must be *built* in that order. See Work Streams. + +--- + +## 3. Work Streams — parallel subagent assignments (build against contracts, not merged code) + +A WS is a **self-contained build assignment** that one subagent can take end-to-end *right +now*, **in parallel with every other stream**, even though the feature's e2e behavior can't +be exercised until upstream WPs land. The lane tree (§2) is a merge topology; the WP DAG (§1) +is a runtime dependency graph. Neither is a build schedule — **all seven streams can be in +flight simultaneously** if each builds against an agreed *contract* rather than against the +other's merged code. + +**What makes that possible — freeze the inter-package contracts first (WS-PRE):** + +- `ConnectionsGatewayInterface` (WP0 ↔ WP3/WP5) — the shared-connection service signatures. +- `TriggersGatewayInterface` incl. `create_subscription` (WP1 ↔ WP3) — the adapter surface. +- `resolve_target_fields(template, context)` (WP2 ↔ WP4) — the resolver signature + the + `{event, subscription, scope}` context shape. +- The subscription/delivery **DTOs** and the `/triggers/*` **route+payload shapes** + (WP3 ↔ WP4/WP6, WP1 ↔ WP5). + +These are small, decidable up front (they're already specified across `mapping.md`, +`mimics.md`, and §4 here). Once frozen, a downstream stream codes against the interface and +**mocks/stubs the dependency in its own unit tests**; the real wiring + e2e test happens when +the dependency merges into its WL ancestor. + +```text +contracts frozen (WS-PRE) + ├─ WS0 WP0 connection extract ┐ + ├─ WS1 WP1 catalog + adapter │ all seven run concurrently; + ├─ WS2 WP2 resolver → SDK │ each subagent builds its WP to a + ├─ WS3 WP3 subscriptions │ complete, unit-tested PR against the + ├─ WS4 WP4 ingress + dispatch │ frozen contracts + stubs for upstream + ├─ WS5 WP5 web catalog/connections │ + └─ WS6 WP6 web subscriptions/deliv. ┘ + → e2e tests light up as WLs merge bottom-up +``` + +What each stream stubs until its dep is real (everything else it owns outright): + +| Stream | Builds | Stubs (frozen contract) until dep merges | +|--------|--------|-------------------------------------------| +| WS0 | shared connections service + migration | — (root) | +| WS1 | catalog + `ComposioTriggersAdapter` | — (root; live Composio creds for the real test) | +| WS2 | resolver move + webhooks repoint | — (root; webhooks suite is the proof) | +| WS3 | subscription/delivery tables + CRUD | `ConnectionsGatewayInterface` (WP0), `TriggersGatewayInterface` (WP1) | +| WS4 | ingress + dispatch | subscription DTO/DAO (WP3), `resolve_target_fields` (WP2) | +| WS5 | catalog/connections UI | catalog API (WP1), `/…/connections` (WP0) — mocked HTTP | +| WS6 | subscription/deliveries UI | `/triggers/subscriptions` + `/deliveries` API (WP3) — mocked HTTP | + +So the streams are assigned to subagents by **area** and run fully in parallel — api (0,1,3,4), +sdk+webhooks (2), web (5,6) — with the contract freeze (WS-PRE) as the one thing that must +happen before fan-out. The only sequential constraint left is *when e2e (not unit) tests can +pass*, and that follows the WL merge order automatically. + +--- + +## 4. Work packages (detail) + +Each WP lists scope, the gap items it closes, dependencies, and the acceptance bar. "AC" +follows the house rule: ungated endpoints get acceptance tests in **both** editions (OSS +basic account, EE inline business+developer account) — see `feedback_oss_ee_test_accounts`. + +### WP0 — Connection extract (A2-2) · WL0 root (anchor `main`) · WS0 + +Move the provider connection out of `/tools` into the shared, routerless `connections` +domain, leaving the `/tools/connections` contract byte-for-byte unchanged. + +- **Closes:** C1, C2, C3, C4, C5, C6 (and lands the C7 *rule* in code). +- **Scope:** + - Rename `tool_connections` → `gateway_connections` (+ `uq_`/`ix_`); rename-only (no data + transform). Author the revision **once in the shared `core_oss` chain** (rooted + `oss000000000`, version table `alembic_version_oss`), which runs in **both** editions — + EE ships the `oss/` tree and runs it from there (no copy in `core_ee`). **Not** the + parked legacy `core` tree (frozen at `park00000000`, where `tool_connections` was + originally added) and **not** `core_ee` (that chain is EE-only divergence; + `gateway_connections` is shared schema). See + `docs/designs/oss-ee-convergence/migration-chains-and-edition-switch.md`. + - Create `core/gateway/connections/` (service + DAO + `ConnectionsGatewayInterface`) and + `dbs/postgres/gateway/connections/` (DBE + DAO + mappings). **No router.** + - Move the Composio auth verbs (initiate/status/refresh/revoke) out of + `ComposioToolsAdapter` into the shared connection adapter. + - Repoint `ToolsService` connection management at the shared service; the + `/tools/connections` and `/callback` handlers now delegate. Fix the ~4 `tool_connections` string refs + (`dao.py:72` error match, `router.py:160` operation_id). + - Implement the **cross-domain revoke rule** (C7): revoke affects all consumers; expose a + "used by" usage read. (No trigger consumer exists yet — this is the rule + the seam.) +- **Functional deps (WP):** none (a root). +- **Lane (WL):** `WL0`, anchored on `main` — the tree root. +- **Stream (WS):** `WS0` — api area; a root, no stubs; runs in parallel with all streams. +- **Decision to lock first:** cross-domain revoke rule (gap C7). +- **AC:** every existing `/tools/connections` test passes **unchanged** (the contract-frozen + invariant); migration up/down clean on both editions; connect/refresh/revoke still work + end-to-end via `/tools/connections`. +- **Risk:** this is the one PR that edits shipped tools code. Keep it a pure refactor + + rename — no behavior change visible at `/tools`. Largest blast radius; review first. + +### WP1 — Triggers skeleton + events catalog + adapter · WL1 (anchor WL0) · WS1 + +Stand up the triggers domain, the read-only events catalog, and the triggers adapter. + +- **Closes:** E1, E2, E3, E4 (and resolves E5). +- **Scope:** + - Domain skeleton `apis/fastapi/triggers/`, `core/triggers/`, `dbs/postgres/triggers/` + (mirror tools layout). + - `ComposioTriggersAdapter` (own httpx client; `triggers_types`, + `trigger_instances/...`) behind `TriggersGatewayInterface`. + - Events catalog: `/triggers/catalog/.../integrations/{i}/events/{event_key}` returning + the event `trigger_config` schema. + - Wiring block in `entrypoints/routers.py` next to tools; built only when + `env.composio.enabled`. + - **Verify exact v3 REST paths against the live OpenAPI spec (E5).** +- **Functional alone:** yes — browse the catalog, fetch a config schema. Read-only, no + connection, no subscription. +- **Functional deps (WP):** none in-feature (uses `env.composio`, not the connection). A + root in the §1 DAG. +- **Lane (WL):** `WL1`, anchored on `WL0` (no functional need for WP0 — anchored here only + to keep the tree linear and make WL1 an ancestor of WL3/WL5). +- **Stream (WS):** `WS1` — api area; a root, no stubs (live Composio creds for the real + test); runs in parallel. +- **AC (both editions):** browse providers/integrations/events; fetch one event's config + schema; catalog empty/disabled when `env.composio` unset. + +### WP2 — Resolver promotion (SDK + webhooks) · WL2 (anchor WL1) · WS2 + +Promote the mapping resolver to the SDK under a neutral name so triggers and webhooks both +consume it without a cross-domain import. A complete, testable change on its own — its +**live consumer today** is webhooks, independent of triggers entirely. + +- **Closes:** M1. +- **Scope:** move `resolve_payload_fields` (`core/webhooks/delivery.py:95`) to + `agenta.sdk.utils.resolvers` as **`resolve_target_fields`** (next to `resolve_json_selector`); + update the webhooks call site to the new name. Pure move + rename, no behavior change. +- **Functional alone:** yes — webhooks delivery resolves payloads through the relocated + resolver; its suite is the proof. +- **Functional deps (WP):** none in-feature. A root in the §1 DAG. +- **Lane (WL):** `WL2`, anchored on `WL1` — *not* a functional need; placed on the line to + WL4 so the resolver is a structural ancestor of WP4 (the one consumer that needs it), + removing the cross-branch merge-order edge. +- **Stream (WS):** `WS2` — sdk+webhooks area; a root, no stubs (webhooks suite is the proof); + runs in parallel. +- **AC:** existing webhook delivery tests pass unchanged against the renamed/relocated + resolver. + +### WP3 — Subscriptions + deliveries · WL3 (anchor WL2) · WS3 + +The two-table heart of the domain. **Hard-depends on `gateway_connections` existing** (the +subscription FK). Functional as **subscription CRUD** before any dispatch exists. + +- **Closes:** S1, S2, S3, S4, S5. +- **Scope:** + - `subscriptions` table (FlagsDBA, DataDBA): `ti_*`, `trigger_config`, `inputs_fields`, + destination `references`/`selector`, workflow ref, **FK → `gateway_connections`**. + - `deliveries` table: resolved `inputs`, workflow `references`, `result`/`error`, plus the + `metadata.id` dedup column (I4). + - DBA mixins for both (mirror `dbs/postgres/webhooks/dbas.py`). + - Migration authored once in the shared `core_oss` chain (both editions, per WP0's note). + - Subscription CRUD `/triggers/subscriptions/` · `/query` · `/{id}` · `/{id}/refresh` · + `/{id}/revoke`, creating/disabling/deleting the Composio `ti_*` through the adapter and + referencing a shared connection (deleting a subscription must **not** revoke the + connection — C7). + - Delivery read routes `/triggers/deliveries` · `/{id}` · `/query`. +- **Functional alone:** yes — create/list/disable/delete a subscription (and its Composio + `ti_*`), read the deliveries table. The standing-watch lifecycle works end-to-end even + though nothing is dispatching into it yet. +- **Functional deps (WP):** **WP0** (FK → `gateway_connections`) **and** **WP1's adapter** + (`create_subscription` builds the `ti_*` — the adapter, not the catalog routes). A fan-in + in the §1 DAG. +- **Lane (WL):** `WL3`, anchored on `WL2`; both functional parents are tree ancestors (WL0 + and WL1 sit above WL2), so neither needs merge-order coordination and there is no stub. +- **Stream (WS):** `WS3` — api area; runs in parallel, stubbing `ConnectionsGatewayInterface` + (WP0) and `TriggersGatewayInterface` (WP1) against their frozen contracts until those merge. +- **Decision to lock first:** idempotency store (I4 — column on `deliveries`); default + mapping + validation posture (M8). +- **AC (both editions):** create a subscription on a shared connection bound to a workflow; + list/disable/delete; deleting it leaves the connection intact; deliveries list returns + rows. + +### WP4 — Ingress + dispatch · WL4 (anchor WL3) · WS4 + +Close the loop in **one** functional unit: an inbound event is received, verified, scoped, +resolved, and acted on. Ingress lives here (not as its own lane) because a verify-and-park +endpoint isn't functional on its own — the receive path only becomes real once it dispatches. + +- **Closes:** I1, I2, I3, I4, I5, I6, M2, M3, M4, M5, M6, M7, M9; consumes M1. +- **Scope (ingress half):** + - `POST /triggers/composio/events/` reading raw body before parse (mimic billing). + - HMAC-SHA256 verify over `{id}.{ts}.{body}` with `COMPOSIO_WEBHOOK_SECRET`; 401 bad sig; + 200 no-op when secret unset; add `COMPOSIO_WEBHOOK_SECRET` to `env`. + - Recover `project_id` from `metadata.user_id`; route `metadata.trigger_id` → local + subscription; 200-skip unknown/disabled; optional `target`-style env guard (I5). + - One-time project webhook-URL registration with Composio (I6). +- **Scope (dispatch half):** + - Resolve `inputs_fields` via `resolve_target_fields` against `{event, subscription, scope}` + with `TRIGGER_EVENT_FIELDS` (M2, M3) into `data.inputs` only. + - Build the `WorkflowServiceRequest`: destination from the stored workflow `references`/ + `selector` (M4); call `WorkflowsService.invoke_workflow` (M5). + - **System-initiated identity** (M6) — run as a resolved project-system `user_id`. + - **Async dispatch** (M7) — ack-fast + enqueue; ingress returns 2xx promptly. + - Real `metadata.id` dedup against `deliveries` (I4); write a delivery row per event with + outcome; dispatch retry policy (M9). +- **Functional alone:** yes — this is the first PR where a signed inbound event invokes a + workflow and lands a delivery row. The whole feature becomes usable here. +- **Functional deps (WP):** **WP3** (subscriptions + deliveries to read/write) **and** **WP2** + (imports `resolve_target_fields`). A fan-in in the §1 DAG. +- **Lane (WL):** `WL4`, anchored on `WL3`; WP2 (`WL2`) is a tree ancestor of WL3, so the + resolver import is structural — no merge-order edge, no old-location import. +- **Stream (WS):** `WS4` — api area; runs in parallel, stubbing the subscription DTO/DAO (WP3) + and `resolve_target_fields` (WP2) against their frozen contracts until those merge. +- **Decisions to lock first:** webhook-URL registration (I6), sync-vs-async (M7), system + `user_id` (M6), retry policy (M9). +- **AC (both editions):** forged signature → 401; unset secret → 200 no-op; signed event + for a known subscription → bound workflow invoked with the mapped inputs; duplicate + `metadata.id` → single invocation; bad mapping / missing workflow → a `deliveries` error + row (no workflow trace), still 2xx to the provider. + +### WP5 — Web: catalog + connections UI · WL5 (anchor WL1) · WS5 + +The browse half of the FE: providers/integrations/events and the connection list. + +- **Closes:** F1 (catalog/connect part), F2. +- **Scope:** "Triggers" entry on a connected integration — browse events and their config + schema (WP1 catalog API); show connections via `/triggers/connections`; handle the + **overlapping connection reads** across `/tools/connections` and `/triggers/connections` + (same rows, F2). +- **Functional alone:** yes — browse events and see connections against a merged WP1, even + before subscriptions exist. +- **Functional deps (WP):** **WP1** (catalog API) **and** **WP0** (the `/…/connections` view + over `gateway_connections`). A fan-in in the §1 DAG. +- **Lane (WL):** `WL5`, anchored on `WL1`; WP0 (`WL0`) is a tree ancestor, so both deps are + covered. (Sibling of WL2 under WL1 — fan-out off WL1 is fine.) +- **Stream (WS):** `WS5` — web area; runs in parallel, mocking the catalog (WP1) and + `/…/connections` (WP0) HTTP against their frozen shapes until those merge. +- **AC:** browse a connected integration's events; the same connection appears under both + tools and triggers without a second connect. + +### WP6 — Web: subscriptions + deliveries UI · WL6 (anchor WL3) · WS6 + +The management half of the FE: create/manage subscriptions and view deliveries. + +- **Closes:** F1 (subscribe part), F3. +- **Scope:** create a subscription (pick event + bind workflow + mapping), list / disable / + delete (WP3 subscription API); deliveries audit view (`/triggers/deliveries`, F3 — + deferrable past v1). +- **Functional alone:** yes — create and manage subscriptions against a merged WP3; a new + subscription simply shows no deliveries until WP4 dispatch lands. +- **Functional deps (WP):** **WP3** only (the `/triggers/subscriptions` + `/triggers/deliveries` + API). Independent of WP4 — the management UI doesn't need dispatch to exist. +- **Lane (WL):** `WL6`, anchored on `WL3` (sibling of WL4 — WL3 fans out to both). +- **Stream (WS):** `WS6` — web area; runs in parallel, mocking the WP3 HTTP surface + (`/triggers/subscriptions` and `/triggers/deliveries`) against its frozen shape until WP3 + merges. +- **AC:** create a workflow-bound subscription; list/disable/delete it; deliveries view + renders (empty until WP4). + +--- + +## 5. The three views, side by side + +Same seven units, the three edge sets together. Read across a row to see how one unit looks +in each lens. + +| k | Unit | Closes | WP — functional deps | WL — anchor | WS — area · stubs until dep merges | +|---|------|--------|----------------------|-------------|-------------------------------------| +| 0 | connection extract | C1–C7 | — | `main` | api · — | +| 1 | catalog + adapter | E1–E5 | — | WL0 | api · — | +| 2 | resolver → SDK | M1 | — | WL1 | sdk+webhooks · — | +| 3 | subscriptions + deliveries | S1–S5 | WP0, WP1 | WL2 | api · stubs ConnectionsGW (WP0), TriggersGW (WP1) | +| 4 | ingress + dispatch | I1–I6, M2–M9 | WP3, WP2 | WL3 | api · stubs subs DTO (WP3), resolver (WP2) | +| 5 | web catalog/connections | F1, F2 | WP1, WP0 | WL1 | web · mocks catalog (WP1), /connections (WP0) | +| 6 | web subscriptions/deliveries | F1, F3 | WP3 | WL3 | web · mocks /subscriptions+/deliveries (WP3) | + +The WL anchors form the tree of §2; every WP fan-in (rows 3, 4, 5) is covered because the +non-anchor parent is a tree ancestor. The WS column is the parallel-subagent view of §3 — all +seven build concurrently against frozen contracts (WS-PRE), stubbing the listed dep until it +merges; e2e tests light up in WL merge order. + +--- + +## 6. Risks & sequencing notes + +- **WP0 is the only PR that touches shipped tools code.** Keep it a pure refactor+rename + with the `/tools/connections` contract frozen; it is the tree root, so it is reviewed and + merged first regardless. A regression here hits live tools. +- **GitButler stacking caveat (from `vibes/AGENTS.md`):** keep the WL tree a true GitButler + stack (`--anchor`); do **not** sync branches by merging them into each other — a + merge-based series can collapse to a single addressable tip on unapply/re-apply. Snapshot + (`but oplog snapshot`) before risky stack surgery. +- **Stacked PR bases follow the WL anchors:** each PR sets `--base` to its anchor branch + (e.g. `wp3` `--base wp2`, `wp4` `--base wp3`, `wp5` `--base wp1`, `wp6` `--base wp3`) so + each shows only its own diff. +- **No merge-order coordination needed.** Because every functional dep is a WL ancestor (§2), + there is no "merge X before Y" rule to remember — the tree enforces it. (This is why the + tree linearizes WP2 and WP5 under WL1 rather than running them as free parallel lanes.) +- **Decisions that gate code** (from `gap.md` §3) close at the head of the WP that needs them + — revoke rule before WP0; REST paths (E5) before WP1's adapter; idempotency + mapping + default before WP3; async + identity + retry + URL-registration before WP4. +- **Build order ≠ lane order.** The WL tree is a merge topology; the WS view (§3) is parallel + build assignments against frozen contracts. A branch deep in the tree (e.g. WP4) can be in + active development while an ancestor (e.g. WP1) is still in review — GitButler lets you push + fixes mid-stack, and the contract freeze lets the subagent build before WP1 merges. +- **Contract freeze (WS-PRE) is the one true prerequisite.** Parallelism depends on the + inter-package interfaces (§3) being fixed before fan-out; a contract change after fan-out + forces a re-sync across the dependent streams. Lock them with the gate decisions above. diff --git a/docs/designs/gateway-triggers/proposal.md b/docs/designs/gateway-triggers/proposal.md new file mode 100644 index 0000000000..b364c699c1 --- /dev/null +++ b/docs/designs/gateway-triggers/proposal.md @@ -0,0 +1,236 @@ +# Gateway Triggers — Proposal + +## Summary + +Add **triggers** to the gateway as a first-class, standalone concept, symmetric to the +existing gateway **tools**. A trigger lets a project subscribe to an *inbound* event +from a connected provider (new Gmail message, new GitHub commit, new Slack message) and, +when that event fires, **invoke an Agenta workflow** with the event as input. Triggers +are a peer top-level domain (`/triggers`, alongside `/tools`) with their own router, +service, DAO, and `subscriptions` table. Provider connections (`ca_*`) are **shared**: an +extracted `connections` domain (table `gateway_connections`, renamed from +`tool_connections`) backs both tools and triggers, so a provider is connected once and +used from both (decision **A2-2**; see [Alternatives](#alternatives-considered)). + +The guiding analogy: + +```text +Agenta events ──▶ user endpoints (outbound; the existing `webhooks` domain) +Composio triggers ──▶ Agenta workflows (inbound; this design) +``` + +So a trigger is the inbound dual of an event subscription: where the `webhooks` domain +pushes Agenta-internal events *out* to a customer's URL, a gateway trigger pulls a +provider event *in* and runs it through an Agenta workflow. Triggers are their **own +domain concept** — not the outbound `webhooks` domain, and not workflow hooks. +See "Non-goals". + +## Why + +Tools answer "let the model *do* something in a provider." Triggers answer the inverse: +"let a provider *tell Agenta* something happened, and run an Agenta workflow on it." +Together they make the gateway bidirectional. This is the symmetric counterpart to the +existing outbound `webhooks` domain: Agenta events flow *out* to user endpoints; provider +triggers flow *in* to Agenta workflows. The `/tools` vertical already proved the +gateway-via-Composio pattern end to end; triggers replicate that proven structure in a +standalone domain for the inbound direction. + +## Goals + +1. **Event catalog** — browse the **events** a connected integration exposes, including + each event's required `trigger_config` schema. Symmetric to the tools action catalog. +2. **Subscription lifecycle** — on a (shared) connection, create / enable / disable / + delete many *subscriptions*, each a standing watch on one event bound to one workflow. + Persisted in the triggers domain's own `subscriptions` table; connection auth lives in + the shared `connections` domain. +3. **Ingress** — one server-owned, signature-verified inbound endpoint that receives + Composio's webhook deliveries, maps each event to the owning project + trigger + record, and dedups redeliveries. +4. **Dispatch to a workflow** — when a verified event arrives, invoke the Agenta + workflow bound to that subscription, passing the event as input. This is the + point of the feature: `Composio event → Agenta workflow`, mirroring + `Agenta event → user endpoint`. The binding (`subscription → workflow ref`) is + stored on the subscription record; dispatch calls the existing + `WorkflowsService.invoke_workflow(project_id, user_id, request)` seam + (`core/workflows/service.py:1698`). +5. **Peer `/triggers` domain alongside `/tools`** — triggers get their own top-level + endpoint (not nested under `/tools`), their own router, service, DAO, DTOs, and their + own `subscriptions` table. `/tools` for outbound actions, `/triggers` for inbound + events. Triggers' event-catalog, subscription, and dispatch code is separate from + tools'. +6. **Shared provider connections (decision: A2-2)** — the provider connection (`ca_*`) is + a **gateway-level primitive**, not a per-feature resource: one Composio connected + account is the same account whether a tool calls it or a trigger watches it. It is + extracted into a shared `connections` domain (service + DAO + `gateway_connections` + table, renamed from `tool_connections`) that has **no router of its own**. The HTTP + surface stays per-domain — `/tools/connections` and `/triggers/connections` — both + delegating to the shared service over the same rows. **Connect a provider once; use it + from both tools and triggers.** Tools' connection auth is repointed at the shared + service; the `/tools/connections` HTTP contract is unchanged. See + [Alternatives considered](#alternatives-considered) for the rejected fully-separate + option (B). +7. **Provider-agnostic shape** — model the shared connections adapter and the triggers + adapter behind ports so a future non-Composio provider drops in without touching + routers or services. + +## Non-goals + +- **Not the outbound `webhooks` domain.** That domain (Agenta → customer URLs, driven by + internal `EventType`s, with its own subscriptions/deliveries/retries) stays exactly as + is. Triggers are inbound (provider → Agenta) and are a separate domain with their own + router, service, and table. We do **not** merge them, and we do **not** route trigger + ingress through the webhooks subscription/delivery machinery in v1. +- **Not workflow hooks.** Workflow lifecycle hooks are an unrelated mechanism; triggers + do not extend, replace, or depend on them. +- **Workflow invocation is the only v1 consumer.** A trigger binds to exactly one + Agenta workflow and invokes it on each event. Other downstream consumers (evaluations, + queues, re-emitting as an internal Agenta event for the outbound `webhooks` domain) are + deliberately out of scope for v1 — the dispatch step is kept narrow: resolve the bound + workflow and call `invoke_workflow`. +- **No new workflow execution path.** Triggers invoke workflows through the existing + `WorkflowsService` seam; we do not build a parallel runner. +- **No custom-OAuth ingress registration** (registering Composio's ingress URL on a + customer's own OAuth app). Managed-auth only for v1. +- **No polling fallback we own.** Composio handles provider polling for polling-type + triggers; we only consume its single normalized webhook. +- **No SDK dependency.** `httpx` direct calls, same as tools. +- **No EE-only gating beyond what tools already have.** Triggers ship wherever tools do. + +## Shape of the solution (high level) + +```text +Provider ──event──▶ Composio ──signed webhook──▶ POST /triggers/composio/events/ + │ verify HMAC (raw body) + │ route metadata.trigger_id → local record + │ recover project from metadata.user_id + │ dedup on metadata.id + ▼ + resolve bound workflow ref on the record + ▼ + WorkflowsService.invoke_workflow( + project_id, user_id, request=event-as-input) + +Project ──▶ POST /triggers/connections/ (connect provider, OAuth) ──┐ shared connection (ca_*) + (or /tools/connections — same shared service + rows) │ (also usable from tools) + ──▶ POST /triggers/subscriptions/ (pick event + bind workflow) ├─▶ services ─▶ Composio v3 + ──▶ GET /triggers/catalog/.../events/... (events) ┘ (one ca_* ; many ti_* per ca_*) +``` + +Terminology (see `mimics.md`): catalog leaf = **event** (≈ tools **action**). The created +state is two records with different owners and cardinality: + +- **connection** — durable provider auth (`ca_*`), one per (project, provider, + integration). A **gateway-level** resource shared by tools and triggers, in the + `connections` domain. The inbound/outbound-neutral evolution of today's tool connection. +- **subscription** — a standing watch on one event (`ti_*` + `trigger_config` + bound + workflow), FK → connection. Owned by the triggers domain. The inbound dual of a + **webhook subscription**. + +Why split connection from subscription: a Composio connected account (`ca_*`) backs +**many** trigger instances (`ti_*`) — Gmail "new message" and "new starred message" share +one auth. Tools already separates durable auth from per-use detail (a connection holds +only auth; the action + arguments arrive per call). Triggers is the first domain that must +*persist* per-event detail, so the connection/subscription split makes the +1-connection → many-subscriptions cardinality explicit (connect once, subscribe many). + +Why share connections across domains (A2-2): `ca_*` is one real account regardless of +consumer; two rows for it would encode a lie and force a second OAuth consent. So: + +- **`connections` (shared domain, no router)** — `core/gateway/connections/` + + `dbs/postgres/gateway/connections/`. Owns OAuth initiate / callback / refresh / revoke + and the `gateway_connections` table (renamed from `tool_connections`; already + domain-neutral). Its Composio **auth** adapter implements a `ConnectionsGatewayInterface`. + **No `apis/fastapi/gateway/connections/` router** — the HTTP surface is the per-domain + `/tools/connections` and `/triggers/connections`, both delegating to this one service + over the same rows. +- **`triggers` (peer domain)** — `apis/fastapi/triggers/`, `core/triggers/`, + `dbs/postgres/triggers/`. A **two-table** domain mirroring webhooks' subscription + + delivery pair: + - `subscriptions` — project-scoped, FlagsDBA (enabled/valid), DataDBA with `ti_*`, the + mapping (`inputs_fields`), the destination (`references`/`selector`), and the **bound + workflow ref**; FK → shared connection. + - `deliveries` — one audit row per inbound event dispatched (resolved `inputs`, workflow + `references`, `result`/`error`); the audit + retry surface, mirroring + `webhook_deliveries`. + + Plus the event catalog, ingress, and dispatch. Three routers under `/triggers`: + - `/triggers/connections` — delegates to the shared `connections` service (the triggers + view onto `gateway_connections`). + - `/triggers/subscriptions` — the standing watches (own `subscriptions` table). + - `/triggers/deliveries` — the dispatch audit log (own `deliveries` table). + + (Plus the catalog routes and the `/triggers/composio/events/` ingress.) Its Composio + **triggers** adapter implements a `TriggersGatewayInterface` (`list_events`, `get_event`, + `create_subscription`, `set_subscription_status`, `delete_subscription`). It depends on + the shared `connections` service for auth and on `WorkflowsService` for dispatch. +- **`tools` (existing domain)** — unchanged HTTP contract; its connection auth is + repointed at the shared `connections` service. Keeps actions + execution. +- One provider-namespaced ingress endpoint, **`POST /triggers/composio/events/`**, + with HMAC verification keyed on a `COMPOSIO_WEBHOOK_SECRET`. This follows the + established `{domain}/{provider}/events/` convention — cf. billing's + `/billing/stripe/events/` (`api/ee/src/apis/fastapi/billing/router.py:106`), which + likewise reads the raw body and verifies a provider signature + (`stripe.Webhook.construct_event` with `env.stripe.webhook_secret`). Namespacing by + provider leaves room for a future `/triggers/{provider}/events/` without collision. + +## Success criteria + +- A project can connect Gmail **once** (a shared `gateway_connections` row), browse + Gmail's **events**, create a "new message" **subscription bound to a chosen Agenta + workflow** (and more subscriptions on the same connection without re-auth), and have + that workflow invoked with the event payload when a new message arrives. +- A Gmail already connected for tools is usable by triggers without reconnecting, and + vice-versa; the same connection shows in both `/tools/connections` and + `/triggers/connections` (same shared rows). +- The invocation is project-scoped and authenticated through the existing + `invoke_workflow` path (no new execution route). +- Disabling/deleting a subscription stops delivery and removes the Composio trigger + instance, without touching the shared connection. +- Forged or replayed deliveries are rejected (signature + dedup). +- No change to the outbound `webhooks` domain or to the existing `/tools` HTTP contract. + +## Risks / decisions to lock before build + +- **Exact Composio v3 trigger REST paths** (verify against live OpenAPI; SDK names are + stable). +- **How the project webhook URL is registered** (API vs dashboard) and whether one URL + per Composio project forces all projects through one ingress (it does — routing is + ours). +- **Event → workflow mapping** — worked out in [`mapping.md`](mapping.md): destination is a + workflow `references`/`selector` (the `/retrieve` shape); the `inputs_fields` template + (webhooks' `payload_fields`, retargeted) resolves the inbound event into + `WorkflowServiceRequest.data.inputs` via the reused selector resolver. Open sub-points: + the default mapping, schema-validation against the bound workflow, and what `user_id` a + system-initiated invocation runs as (no human in the loop). +- **Sync vs async dispatch** — invoke inline in the ingress request, or enqueue and ack + fast so Composio's webhook doesn't time out / retry. Leaning async. +- **Idempotency store** for `metadata.id` dedup (table column vs cache). +- **Cross-domain revoke rule (consequence of A2-2).** Because a connection is shared, + revoking a `ca_*` affects every consumer (tools actions + trigger subscriptions on it). + Lean: **revoke-for-everyone + show usage** ("used by tools / used by N subscriptions") + rather than cross-domain reference-counting. Deleting a subscription must *not* revoke + the shared connection. The FE must expect overlapping reads across the three connection + surfaces. This rule is the main new behavior A2-2 introduces. +- **`gateway_connections` migration.** Rename `tool_connections` → `gateway_connections` + (+ its `uq_`/`ix_` constraints); no data transform (table is already domain-neutral). + Repoint tools' connection auth (~4 references) at the shared `connections` service. The + `/tools/connections` contract stays frozen. + +## Alternatives considered + +### B — fully separate connections (rejected) + +`tool_connections` stays as-is; triggers gets its own `trigger_connections` (a mirror). +Zero migration, zero cross-domain coupling, no shared-lifecycle rule. + +**Why rejected:** it buys nothing for the user and encodes a falsehood. A Composio +connected account is one real account; modeling it as two rows forces the user to connect +the same provider **twice** (two OAuth consents, two "Gmail connected" states) for tools +vs. triggers, indefinitely. B is the smaller raw diff, but the cost is paid forever in +duplicate consent. A2-2 was chosen because the migration turned out cheap (`tool_connections` +is recent, ~4 references, and already provider-agnostic) — so the only real added cost of +A2-2 over B is the cross-domain revoke rule above, which is small and worth it. + +A2-1 (shared `gateway_connections` table but **separate rows per domain**) was also +rejected: it pays A2's migration cost while still forcing connect-twice — all of the cost, +none of the benefit. diff --git a/docs/designs/gateway-triggers/research.md b/docs/designs/gateway-triggers/research.md new file mode 100644 index 0000000000..35235183de --- /dev/null +++ b/docs/designs/gateway-triggers/research.md @@ -0,0 +1,403 @@ +# Gateway Triggers — Research + +Status quo, internal and external, for adding **triggers** (inbound provider events) +to the gateway alongside the existing **tools** (outbound action calls). + +--- + +## 0. Terminology and the shared-connection decision + +Three nouns, drawn from existing domains so the whole thing reads familiar: + +| Concept | Owner | Tools | Webhooks | Triggers | What it is | +|---------|-------|-------|----------|----------|------------| +| catalog leaf | per-domain | **action** | — | **event** | callable action vs. watchable event | +| provider auth | **shared** `connections` | connection (`ca_*`) | — | connection (`ca_*`) | one per (project, provider, integration), via OAuth | +| standing event watch | triggers | — | subscription | **subscription** (`ti_*` + config + workflow) | many per connection | + +Catalog hierarchy maps cleanly: + +```text +tools: providers / integrations / actions +triggers: providers / integrations / events +``` + +The created state is two records with **different owners**: + +```text +shared: connection (ca_*) ← gateway_connections; used by BOTH tools and triggers +triggers: event (catalog) → subscription (ti_* + trigger_config + workflow) ← FK → connection +``` + +**Why connection and subscription are split, and why the connection is shared (A2-2):** + +- *Split* — a Composio connected account (`ca_*`) backs many trigger instances (`ti_*`): + one Gmail auth serves "new message", "new starred", etc. So a **subscription** (one + standing watch, bound to one workflow) is separate from the **connection** (durable + auth). Connect once, subscribe many. Tools never persisted the per-use record (a tool + call is ephemeral); webhooks never had a connection (no provider to authenticate); + triggers is the first domain needing both. +- *Shared* — `ca_*` is one real account regardless of consumer. Rather than each domain + owning its own copy, the connection is extracted into a **shared `connections` domain** + (`gateway_connections` table, renamed from `tool_connections`; service + DAO, **no + router of its own**), consumed by both tools and triggers. Connect Gmail once → usable + from both. HTTP surface is per-domain — `/tools/connections` and `/triggers/connections` + both delegate to the one shared service over the same rows. + (Decision **A2-2**; rejected alternative **B** — fully separate connections — and full + reasoning in `proposal.md` § Alternatives and `mimics.md`.) + +Composio's own vocabulary ("trigger type", "trigger instance") is kept only when +describing the Composio API itself; in Agenta terms they are an **event** and the +provider-side half of a **subscription**. + +--- + +## 1. External: how Composio triggers work + +Composio's [Triggers](https://docs.composio.dev/docs/triggers) are the mirror image of +its tools. Tools are *outbound* — you call a provider action (`GMAIL_SEND_EMAIL`). +Triggers are *inbound* — a provider emits an event (new Slack message, new GitHub +commit, new Gmail message) and Composio delivers it to you. + +### Core concepts + +| Composio concept | Agenta term | Meaning | Composio ID prefix | +|------------------|-------------|---------|--------------------| +| **Trigger type** | **event** (catalog leaf) | Template defining an event to watch + required config. E.g. `GITHUB_COMMIT_EVENT` needs `owner`, `repo`. Each toolkit exposes its own trigger types. | (slug, e.g. `GITHUB_COMMIT_EVENT`) | +| **Trigger instance** | part of a **subscription** | A trigger type *instantiated* for one user + one connected account, with concrete config. Independently enable/disable/delete. | `ti_*` | +| **Connected account** | part of a **subscription** | The authenticated binding a trigger is scoped to. **A trigger cannot exist without one** — auth comes first. | `ca_*` | + +### Two delivery mechanisms (transparent to us) + +- **Webhook triggers** (Slack, Notion, Asana, Outlook): provider pushes to a + Composio-issued ingress URL in real time. +- **Polling triggers** (Gmail, Google Calendar): Composio polls the provider on a + schedule; with Composio-managed auth the worst-case source→delivery delay is ~15 min. + +Either way, Composio normalizes both into one outbound webhook to **our** subscription +URL. We never talk to the provider directly. + +### Lifecycle (per the docs) + +1. **Subscribe** (once per Composio project): tell Composio the single webhook URL to + deliver all trigger events to. +2. **Discover**: list trigger types for a toolkit; read each type's required `config`. +3. **Create**: create an active trigger instance scoped to a `user_id` + + connected account, with `trigger_config`. +4. **Receive**: events arrive at our subscription URL as HTTP POST; route on + `metadata.trigger_slug`. +5. **Manage**: enable / disable / delete instances. + +### SDK / REST surface + +The Python SDK (`composio.triggers.*`) wraps a REST surface. From the docs and SDK: + +```python +# Discover required config +trigger_type = composio.triggers.get_type("GITHUB_COMMIT_EVENT") +trigger_type.config # JSON Schema of required trigger_config + +# Create an instance (scoped to a user + their connected account) +trigger = composio.triggers.create( + slug="GITHUB_COMMIT_EVENT", + user_id="project_019abc...", + trigger_config={"owner": "composiohq", "repo": "composio"}, +) +trigger.trigger_id # ti_* + +# Local-dev only: SDK-managed subscription (websocket), not for prod +subscription = composio.triggers.subscribe() +@subscription.handle(trigger_id="ti_...") +def handler(data): ... +``` + +REST equivalents (we use `httpx` directly, no SDK — same decision as tools): + +| Operation | REST (v3) | +|-----------|-----------| +| List trigger types for a toolkit | `GET /api/v3/triggers_types?toolkit_slugs={slug}` | +| Get one trigger type (config schema) | `GET /api/v3/triggers_types/{slug}` | +| Create / upsert instance | `POST /api/v3/trigger_instances/{slug}/upsert` (`user_id`, `trigger_config`) | +| Enable / disable instance | `PATCH /api/v3/trigger_instances/manage/{trigger_id}` (`status`) | +| Delete instance | `DELETE /api/v3/trigger_instances/manage/{trigger_id}` | +| List instances | `GET /api/v3/trigger_instances` (filter by `user_id`, `toolkit`) | +| Set project webhook URL | project settings / `POST /api/v3/...webhook` (one-time, dashboard or API) | + +> Exact paths must be confirmed against the live OpenAPI spec during implementation; +> the SDK method names (`get_type`, `create`, `subscribe`) are stable. This is the +> same "verify against live spec" caveat that landed for the tools endpoints. + +### Webhook payload (V3, the default for new orgs) + +```json +{ + "type": "github_commit_event", + "timestamp": "2026-06-18T10:00:00Z", + "data": { /* provider event payload, trigger-type-specific */ }, + "metadata": { + "id": "evt_...", + "trigger_slug": "GITHUB_COMMIT_EVENT", + "trigger_id": "ti_...", + "toolkit_slug": "github", + "user_id": "project_019abc...", + "connected_account": { "id": "ca_...", "status": "ACTIVE" } + } +} +``` + +We route on `metadata.trigger_slug` (which trigger type) and `metadata.trigger_id` +(which instance) → our local trigger record → project scope. +`metadata.user_id` carries our `project_{project_id}` scope verbatim, the same +`user_id` strategy tools already use. + +### Webhook verification + +Composio signs every webhook with **HMAC-SHA256** (svix-style headers), per +[Verifying webhooks](https://docs.composio.dev/docs/webhook-verification): + +- Headers: `webhook-id`, `webhook-timestamp`, `webhook-signature`. +- Signing string: `{webhook-id}.{webhook-timestamp}.{raw-body}`. +- HMAC-SHA256 with the project webhook secret, base64-encoded; compare with + `hmac.compare_digest`. The `webhook-signature` header may carry a `v1,` prefix. + +```python +signing_string = f"{webhook_id}.{webhook_timestamp}.{raw_body}" +expected = base64.b64encode( + hmac.new(secret.encode(), signing_string.encode(), hashlib.sha256).digest() +).decode() +received = signature.split(",", 1)[1] if "," in signature else signature +ok = hmac.compare_digest(expected, received) +``` + +Verification needs the **raw request body** (not the parsed JSON), so the ingress +endpoint must read `await request.body()` before parsing. + +### Tools vs triggers — the symmetry + +| Axis | Tools (built) | Triggers (proposed) | +|------|---------------|---------------------| +| Direction | Outbound (we call provider) | Inbound (provider calls us) | +| Catalog leaf | **action** slug | **event** slug (Composio trigger type) | +| Durable auth record | **connection** (`ca_*`) | **same shared connection** (`gateway_connections`) | +| Per-use record | *(ephemeral tool call)* | **subscription** (`ti_*` + config + workflow), FK → connection | +| Connection routes | `/tools/connections` | `/triggers/connections` (both delegate to the shared service; no `/gateway/connections` route) | +| Per-domain routes | actions, `/call` | events catalog, `/subscriptions`, ingress | +| Config | arguments per call | `trigger_config` per subscription, set once | +| Entry point | `POST /tools/call` | inbound `POST /triggers/composio/events/` | +| HTTP domain | `/tools/*` | independent `/triggers/*` (peer, not nested) | +| Per-event work | synchronous response to caller | invoke the bound Agenta workflow | + +The single most important external fact: **a trigger, like a tool, is a Composio +resource scoped to a connected account.** Tools proved that pattern; triggers reuse the +**same** (shared) connected account and add events + subscriptions on top (see the +shared-connection decision A2-2 +below). + +--- + +## 2. Internal: how tools are integrated today + +The gateway-tools feature is **shipped** (not just designed). Layout follows the +standard domain shape from `api/AGENTS.md`. + +### Layers + +```text +api/oss/src/apis/fastapi/tools/ router.py · models.py · utils.py +api/oss/src/core/tools/ service.py · interfaces.py · dtos.py + registry.py · exceptions.py · utils.py +api/oss/src/core/tools/providers/composio/ adapter.py · catalog.py · dtos.py +api/oss/src/dbs/postgres/tools/ dbes.py · dao.py · mappings.py +``` + +Dependency direction (enforced): `Router → Service → DAOInterface + GatewayInterface → +DAO impl + Adapter impl`. Concrete wiring lives only in `api/entrypoints/routers.py`. + +### Domain layout — three verticals, shared connections (decision A2-2) + +**Decision:** connections are a **gateway-level primitive shared** by tools and triggers; +the trigger-specific state is a peer domain. Three verticals: + +1. **`connections` (shared, extracted)** — owns the provider connection `ca_*`: OAuth + initiate/callback/refresh/revoke and the `gateway_connections` table (renamed from + `tool_connections`). **No router of its own** — the HTTP surface is `/tools/connections` + and `/triggers/connections`, both delegating to this shared service over the same rows. + Code: `core/gateway/connections/`, `dbs/postgres/gateway/connections/` (service + DAO + + table; no `apis/fastapi/gateway/connections/`). +2. **`triggers` (peer to tools)** — owns events catalog, the `subscriptions` **and** + `deliveries` tables (a two-table domain mirroring webhooks' `webhook_subscriptions` + + `webhook_deliveries`), ingress, and dispatch. Depends on the shared `connections` + service for auth and on `WorkflowsService` for dispatch. +3. **`tools` (existing)** — unchanged HTTP contract; connection auth repointed at the + shared `connections` service. + +`/tools` remains the structural blueprint for the trigger-specific code (copy structure, +swap nouns `action → event`); the connections code is *extracted and shared*, not copied. +(Rejected alternative B — fully separate `trigger_connections` — and why, in +[`proposal.md` § Alternatives].) + +What each part is modeled on: + +- **Shared connections** — evolve the existing tool-connection code in place: + `ToolConnectionDBE` / `tool_connections` (`dbs/postgres/tools/dbes.py`) becomes the + `gateway_connections` DBE in the connections domain (already domain-neutral — no + `tool_`-specific columns). The Composio **auth** adapter (`initiate_connection`, + `get_connection_status`, `refresh_connection`, `revoke_connection` from + `ComposioToolsAdapter`) moves to a `ConnectionsGatewayInterface` in the connections + domain. Tools and triggers both consume it. +- **Triggers adapter** — a **new** `ComposioTriggersAdapter` (own httpx client, + modeled on `ComposioToolsAdapter`'s `_get/_post/_delete` + slug mapping) implementing a + `TriggersGatewayInterface` for the trigger REST surface (`triggers_types`, + `trigger_instances/...`). Helpers may be copied or promoted to a shared util. +- **`subscriptions` table** — modeled on `WebhookSubscription` / `webhook_subscriptions` + (`core/webhooks/types.py:116`): project-scoped, FlagsDBA (enabled/valid), carrying the + trigger instance (`ti_*`), the mapping (`inputs_fields`), the destination + (`references`/`selector`), and a FK → `gateway_connections`. Many per connection. +- **`deliveries` table** — modeled on `WebhookDelivery` / `webhook_deliveries` + (`core/webhooks/types.py:156`): one audit row per inbound event dispatched, carrying the + resolved `inputs`, the workflow `references`, and `result`/`error`. The audit + retry + surface — and the only record when dispatch fails before invocation. (See `mapping.md` + §4.3.) +- **Events catalog** — model on the tools catalog; leaf is **events**: + `/triggers/catalog/providers/{p}/integrations/{i}/events/{event_key}`, returning the + event's `trigger_config` JSON Schema (analogue of an action's `input_parameters`). +- **Service / router / DAO** — `TriggersService` (event-catalog browse, subscription CRUD, + ingress, dispatch) models on `ToolsService` + `WebhooksRouter`'s `/subscriptions/...` + shape; depends on its own DAO + triggers adapter + the shared connections service + + `WorkflowsService`. +- **Env** — `env.composio` (`api_key`, `api_url`) read directly; add + `COMPOSIO_WEBHOOK_SECRET`. + +Route map: + +| Surface | Route | Patterned on | +|---------|-------|--------------| +| connections (triggers view) | `/triggers/connections/` · `/query` · `/{id}` · `/{id}/refresh` · `/{id}/revoke` · `/callback` | tools connections (shared service) | +| connections (tools view) | `/tools/connections/...` | same shared service + rows | +| events catalog | `/triggers/catalog/.../integrations/{i}/events/{event_key}` | tools catalog | +| subscriptions | `/triggers/subscriptions/` · `/query` · `/{id}` · `/{id}/test` | webhook subscriptions | +| deliveries | `/triggers/deliveries` · `/{id}` · `/query` | webhook deliveries | +| ingress | `/triggers/composio/events/` | billing `/stripe/events/` | + +(There is **no** `/gateway/connections` route — the shared `connections` domain has no +router; the two views above are its only HTTP surfaces.) + +> Firm decisions: connections is a shared gateway primitive (`gateway_connections`, A2-2); +> `/triggers` is a peer domain owning subscriptions + dispatch; the sanctioned cross-domain +> runtime calls are triggers → connections service (auth) and triggers → +> `WorkflowsService.invoke_workflow` (dispatch). + +> **Consequence — cross-domain revoke.** Because `ca_*` is shared, revoking it affects +> both tools actions and trigger subscriptions on it. Lean: revoke-for-everyone + show +> usage; deleting a subscription must not revoke the connection. Connect once, used +> everywhere — the inverse of the connect-twice cost that rejected option B carried. + +### The workflow dispatch seam + +Dispatch invokes the existing +`WorkflowsService.invoke_workflow(*, project_id, user_id, request: WorkflowServiceRequest)` +(`core/workflows/service.py:1698`). It signs a secret token from the project's +workspace/org, resolves the workflow's service URL from the bound revision, and calls it. +Triggers build a `WorkflowServiceRequest` from the verified event and call this — no new +execution path. The open question is the **event → `WorkflowServiceRequest` mapping** and +what `user_id` a system-initiated (no-human) invocation runs as. + +### The OAuth callback is the closest existing analogue to a webhook ingress + +`GET /tools/connections/callback` (`router.py:785`) already implements the inbound +pattern we need for trigger ingress: + +- Server-owned callback URL with an **HMAC-signed `state` token** (`make_oauth_state` / + `decode_oauth_state`, keyed on `env.agenta.crypt_key`) that recovers `project_id` + without trusting the caller. +- Looks up the local connection by provider-side ID + (`activate_connection_by_provider_connection_id`) and mutates local state. +- Returns a controlled response. + +Trigger ingress is the same shape: verify a signature, recover project scope from the +payload's `user_id`/`trigger_id`, look up the local record, then act. + +### The Stripe webhook is the direct precedent for the ingress route shape + +Billing already has a provider-namespaced, signature-verified inbound webhook at +**`POST /billing/stripe/events/`** (`api/ee/src/apis/fastapi/billing/router.py:106`). It +reads the raw request body and verifies the provider signature via +`stripe.Webhook.construct_event(payload, sig, env.stripe.webhook_secret)`. This sets the +house convention for inbound provider events: `{domain}/{provider}/events/`. Trigger +ingress should follow it as **`/triggers/composio/events/`** (Composio HMAC-SHA256 in +place of Stripe's verifier, keyed on `COMPOSIO_WEBHOOK_SECRET`). Provider-namespacing +also leaves room for a second trigger provider at `/triggers/{provider}/events/`. + +### Connection scoping / `user_id` strategy + +`user_id = str(project_id)` is passed to Composio as the connected-account scope +(`service.py:230`). Every connection and therefore every trigger is implicitly +project-scoped. The webhook `metadata.user_id` echoes this back, so ingress can map an +inbound event to a project with no extra lookup table. + +### Config & wiring + +- `env.composio` (`utils/env.py:507`): `api_key`, `api_url`, `enabled` (key present). +- Wiring (`entrypoints/routers.py:578`): adapter built only when `env.composio.enabled`, + registered under key `composio`, injected into `ToolsService`, mounted via + `ToolsRouter`. Triggers slot into the same three spots. + +### Frontend + +Tools UI lives in `web/packages/agenta-entities/src/gatewayTool`, +`web/packages/agenta-entity-ui/src/gatewayTool`, and +`web/oss/src/components/pages/settings/Tools`. Catalog browse, connect (OAuth popup + +poll), list/delete connections. Triggers extend these surfaces (a "Triggers" tab on a +connected integration). + +--- + +## 3. Internal: the existing **outbound** webhooks domain (do not confuse) + +There is already a `webhooks` domain +(`api/oss/src/core/webhooks/`, `apis/fastapi/webhooks/`). It is **outbound**: Agenta +emits internal `EventType`s (e.g. `TRACES_QUERIED`) to subscriber-registered URLs, with +subscriptions, deliveries, retries (`WEBHOOK_MAX_RETRIES = 5`), and HMAC signing on the +*sending* side. + +This is the inverse of triggers: + +- **webhooks domain** = Agenta → outside world (we sign and send). +- **gateway triggers** = outside world (via Composio) → Agenta (we verify and receive). + +They are complementary and should **stay separate domains**. But there is a real +integration point: an inbound Composio trigger can be re-emitted as an internal Agenta +event, which the existing webhooks domain then fans out to customer subscribers. That +keeps "deliver events to customers" in one place and avoids a second outbound delivery +engine. See `proposal.md` for whether v1 includes that bridge. + +--- + +## 4. Open external unknowns to verify during implementation + +1. **Exact v3 REST paths** for trigger types / instances (`triggers_types`, + `trigger_instances/{slug}/upsert`, `.../manage/{id}`). SDK names are stable; REST + paths must be confirmed against the live OpenAPI spec — same caveat the tools + endpoints carried. +2. **How the project webhook URL is registered** — dashboard-only vs API. Determines + whether we can automate it per-environment or document a manual setup step. +3. **One webhook URL per Composio project** — all trigger events for all + projects/integrations arrive at a single ingress. Fan-out/routing is entirely on us + (route by `metadata.trigger_id` → local record). +4. **Retry / redelivery semantics** from Composio on a non-2xx from our ingress + (affects idempotency requirements — we must dedup on `metadata.id`). +5. **Custom-OAuth toolkits** may require registering the Composio ingress URL on the + provider's own OAuth app (noted in the Composio docs). Out of scope for managed-auth + v1 but flagged. + +## Sources + +- [Triggers | Composio](https://docs.composio.dev/docs/triggers) +- [Using Triggers | Composio](https://docs.composio.dev/docs/using-triggers) +- [Creating triggers | Composio](https://docs.composio.dev/docs/setting-up-triggers/creating-triggers) +- [Verifying webhooks | Composio](https://docs.composio.dev/docs/webhook-verification) +- [Triggers — TypeScript SDK reference | Composio](https://docs.composio.dev/sdk-reference/type-script/models/triggers) +- [Create or update a trigger | Composio API](https://docs.composio.dev/reference/api-reference/triggers/postTriggerInstancesBySlugUpsert) +- Internal: `api/oss/src/core/tools/`, `api/oss/src/apis/fastapi/tools/router.py`, + `api/oss/src/dbs/postgres/tools/`, `api/oss/src/core/webhooks/`, + `vibes/docs/designs/gateway-tools/` diff --git a/docs/designs/gateway-triggers/wp/WL-runbook.md b/docs/designs/gateway-triggers/wp/WL-runbook.md new file mode 100644 index 0000000000..cbc754077a --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WL-runbook.md @@ -0,0 +1,156 @@ +# Work Lanes — runbook (GitButler) + Work Stream launch prompts + +How to create the WL branches in **`vibes/`** and spin up the WS subagents. Nothing +here is executed yet — these are the exact commands and prompts to run at kickoff. + +> **Where this runs:** ALL of this work — code, lanes, and docs — lives in **`vibes/`**, which +> is already on `gitbutler/workspace`. The sibling `application/` checkout is a separate repo +> and **must not be used for this work**. Subagents and `but` commands all operate in `vibes/`. + +## 1. The lane tree (recap from `../plan.md` §2) + +```text +main +└─ WL0 wp0-connections-extract + └─ WL1 wp1-events-catalog --anchor wp0 + ├─ WL2 wp2-resolver-promote --anchor wp1 + │ └─ WL3 wp3-subscriptions --anchor wp2 + │ ├─ WL4 wp4-ingress-dispatch --anchor wp3 + │ └─ WL6 wp6-web-subscriptions --anchor wp3 + └─ WL5 wp5-web-catalog --anchor wp1 +``` + +Every functional dep is a tree ancestor → no merge-order coordination (see plan §2). + +## 2. Create the lanes (run in `vibes/`, already in workspace mode) + +```bash +# take a snapshot first (recovery point) +but oplog snapshot -m "before gateway-triggers lanes" + +but branch new wp0-connections-extract +but branch new wp1-events-catalog --anchor wp0-connections-extract +but branch new wp2-resolver-promote --anchor wp1-events-catalog +but branch new wp3-subscriptions --anchor wp2-resolver-promote +but branch new wp4-ingress-dispatch --anchor wp3-subscriptions +but branch new wp5-web-catalog --anchor wp1-events-catalog +but branch new wp6-web-subscriptions --anchor wp3-subscriptions +``` + +PR bases (each shows only its own diff): `wp1 --base wp0`, `wp2 --base wp1`, +`wp3 --base wp2`, `wp4 --base wp3`, `wp5 --base wp1`, `wp6 --base wp3`. `wp0 --base main`. + +## 3. Docs lane (WL-x) + +The design docs in `vibes/docs/designs/gateway-triggers/**` go to their own lane in +**`vibes/`** (already in `gitbutler/workspace`): + +```bash +# in vibes/ +but branch new gateway-triggers-docs +but rub docs/designs/gateway-triggers gateway-triggers-docs # stage the folder to the lane +but commit gateway-triggers-docs --only -m "" +but push gateway-triggers-docs +gh pr create --head gateway-triggers-docs --base main --title "" --body "<body>" +``` + +Title + body authored with the `write-pr-description` skill — draft in [§5](#5-docs-pr-draft). + +## 4. WS launch prompts (paste after compact) + +**Git/GitButler is ours, not the subagents'.** We (the orchestrator) create the WL branches, +stage files to them, commit, push, and open PRs. A subagent **only writes source + test files +into the working tree** for its WP. It does **not** run `git`, `but`, `gh`, or any +branch/commit/push/PR command. After a subagent reports done, we assign its changes to the +right WL branch and commit them. + +**Subagents ask, they don't guess.** If a frozen contract looks wrong, a decision in the spec +is unresolved (e.g. WP0 revoke rule, WP4 sync-vs-async), or the scope is ambiguous, the +subagent **stops and returns the question** to us — it must not change a frozen contract, +pick an open decision, or expand scope on its own. We answer; it resumes. + +Freeze the **WS-PRE contracts** first (the interface blocks in each `WP{k}-specs.md`). Then +spawn one subagent per stream. Roots (WS0/WS1/WS2) need no stubs; WS3–WS6 build against the +frozen contracts and stub the named deps. + +Each prompt template: + +> You are implementing **WP{k}** of the gateway-triggers feature in the `vibes/` repo +> (working dir `/Users/junaway/Agenta/github/vibes`). **Do not touch the sibling +> `application/` checkout — it must not be used for this work.** +> Read your spec at `vibes/docs/designs/gateway-triggers/wp/WP{k}-specs.md` and the parent +> design docs it links (`../plan.md`, `../gap.md`, `../mapping.md`, `../mimics.md`, +> `../research.md`). +> +> **Do NOT touch git or GitButler.** Do not run `git`, `but`, `gh`, or any branch/commit/ +> push/PR command. Just create and edit the source and test files for WP{k} in the working +> tree. Branching, committing, and PRs are handled by the orchestrator after you report. +> +> Implement only WP{k}'s scope. For any dependency on another WP, code against the **frozen +> contract** in the specs and stub/mock it in tests (do NOT implement the dependency). Follow +> `vibes/api/AGENTS.md` (layering, DTOs, exceptions) and the migration rule in WP0 +> (`core_oss`, not the parked `core` tree). Write acceptance tests in both editions per the +> spec's AC. +> +> **If anything is unresolved — a frozen contract looks wrong, an open decision in the spec +> isn't decided, or scope is ambiguous — STOP and return the question.** Do not change a +> frozen contract, resolve an open decision, or expand scope yourself. +> +> Keep `vibes/docs/designs/gateway-triggers/wp/WP{k}-status.md` updated as you progress (this +> file is fine to edit — it is notes, not git). List the files you changed in your final +> report so the orchestrator can commit them to the right lane. + +| Stream | files land for branch | (anchor, set by us) | stubs against frozen contract | +|--------|----------------------|---------------------|-------------------------------| +| WS0 | wp0-connections-extract | main | — | +| WS1 | wp1-events-catalog | wp0 | — | +| WS2 | wp2-resolver-promote | wp1 | — | +| WS3 | wp3-subscriptions | wp2 | ConnectionsGW (WP0), TriggersGW (WP1) | +| WS4 | wp4-ingress-dispatch | wp3 | Subscription DTO/DAO (WP3), resolver (WP2) | +| WS5 | wp5-web-catalog | wp1 | catalog API (WP1), /connections (WP0) | +| WS6 | wp6-web-subscriptions | wp3 | /subscriptions + /deliveries (WP3) | + +The "branch" / "anchor" columns are **our** bookkeeping for where we commit the subagent's +output — the subagent itself is branch-agnostic and just writes files. Because subagents don't +touch git, two streams whose files don't overlap can run concurrently in the same tree; we +separate their changes onto the right lanes at commit time (`but rub <path> <branch>` then +`but commit <branch> --only`). + +Recommended kickoff: spawn **WS0, WS1, WS2** first (contract-free roots), then WS3–WS6 once +their upstream contracts are confirmed stable. + +## 5. Docs PR draft + +**Title:** `[docs] Plan gateway triggers: research, proposal, and WP/WL/WS breakdown` + +**Body:** + +``` +## Context +We are adding inbound provider events ("triggers") to the gateway as the dual of the +existing outbound webhooks: Composio triggers invoke Agenta workflows, the way Agenta events +already POST to user endpoints. Before writing code we needed the design fixed and the build +broken into parallelizable units. + +## Changes +Adds the gateway-triggers design set under docs/designs/gateway-triggers/: + +- research, proposal, gap, mimics, mapping: the status quo, the goal, the delta, the + parallels to tools/billing/webhooks, and how the webhook payload-mapping mechanism is + reused for event-to-workflow input mapping. +- plan.md: the work seen through three views over the same seven units. Work Packages are the + functional DAG (fan-in allowed). Work Lanes are the GitButler merge tree (one parent per + branch, no fan-in). Work Streams are parallel subagent assignments that build against frozen + inter-package contracts and stub their upstreams. +- wp/: per-package specs (WP{k}-specs.md) and trackers (WP{k}-status.md), plus this runbook + with the exact `but` lane commands and the subagent launch prompts. + +No application code changes. The connection extract (WP0) documents the one migration +subtlety: it lands in the shared core_oss chain, not the parked core tree. + +## Notes +- Lanes are not created yet; this PR is the plan only. +- The migration-chain rule cross-references docs/designs/oss-ee-convergence. +``` + +(Authored per `write-pr-description`: context-first, concrete, no em dashes, no padding.) diff --git a/docs/designs/gateway-triggers/wp/WP0-specs.md b/docs/designs/gateway-triggers/wp/WP0-specs.md new file mode 100644 index 0000000000..c2303b2e80 --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP0-specs.md @@ -0,0 +1,104 @@ +# WP0 — Connection extract (A2-2) + +**Lane** WL0 (root, anchor `main`) · **Stream** WS0 (api) · **Area** api (touches shipped tools) + +Parent docs: [`../plan.md`](../plan.md) §4, [`../gap.md`](../gap.md) §2.1, [`../proposal.md`](../proposal.md) (A2-2), +[`../mimics.md`](../mimics.md) (Triggers vs Tools, Part B). + +## Goal + +Move the provider connection out of `/tools` into a shared, **routerless** `connections` +domain, leaving the `/tools/connections` HTTP contract byte-for-byte unchanged. This is the +FK root: `gateway_connections` must exist before any subscription can reference it. + +## Closes (gap items) + +C1, C2, C3, C4, C5, C6 — and lands the **C7** cross-domain revoke *rule* in code. + +## Scope + +- **Migration** — rename `tool_connections` → `gateway_connections` (+ its `uq_`/`ix_` + constraints); rename-only, **no data transform**. Author the revision **once in the shared + `core_oss` chain** (rooted `oss000000000`, version table `alembic_version_oss`), which runs + in **both** editions — EE ships the `oss/` tree and runs it from there (no copy in + `core_ee`). **Not** the parked legacy `core` tree (frozen at `park00000000`) and **not** + `core_ee` (EE-only divergence; `gateway_connections` is shared schema). See + `application/docs/designs/oss-ee-convergence/migration-chains-and-edition-switch.md`. +- **Domain** — create `core/gateway/connections/` (service + DAO + `ConnectionsGatewayInterface`) + and `dbs/postgres/gateway/connections/` (DBE + DAO + mappings). **No router.** +- **Adapter** — move the Composio **auth** verbs (`initiate_connection`, `get_connection_status`, + `refresh_connection`, `revoke_connection`) out of `ComposioToolsAdapter` into the shared + connection adapter behind `ConnectionsGatewayInterface`. +- **Repoint tools** — `ToolsService` connection management delegates to `ConnectionsService`; + the `/tools/connections` + `/callback` handlers call through it. Fix only the FORCED + `tool_connections` string refs: tablename + `uq_`/`ix_` in `dbs/postgres/tools/dbes.py`, and + the `uq_tool_connections_*` IntegrityError match at `dao.py:72`. **B2: do NOT rename + `operation_id="query_tool_connections"` at `apis/fastapi/tools/router.py:160`** — it is part + of the frozen `/tools` OpenAPI contract; the table rename does not require touching it. +- **C7 rule (B3)** — `revoke_connection` keeps today's **local-only** behavior verbatim + (`is_valid=False` on the row; **no** provider call, **no** cascade — provider revoke stays + on DELETE). Because tools and triggers read the **same** `gateway_connections` row, that one + flag IS the cross-domain effect ("revoke-for-everyone" via the shared row, not a new provider + call). C7 additionally ships the `usage()` read ("used by tools / N subs") + the seam. + Subscription delete must **not** revoke the connection. + +## Contracts this WP freezes (consumed by WS3, WS5 — freeze in WS-PRE) + +**B1 = Option A (full extract, no leaks).** Two layers, two names — mirroring how tools is +built (`ToolsGatewayInterface` adapter + `ToolsService`). WS3/WS5 freeze against +**`ConnectionsService`**, not the adapter port. **Nothing in `connections` imports from +`tools`** (no leak): `ToolsService` depends on `ConnectionsService`, never the reverse. + +```text +# SERVICE — project-scoped, owns gateway_connections, returns domain DTOs. WS3/WS5 consume THIS. +ConnectionsService: + initiate_connection(*, project_id, provider, integration, ...) -> Connection + get_connection_status(*, project_id, connection_id) -> Status + refresh_connection(*, project_id, connection_id) -> Connection + revoke_connection(*, project_id, connection_id) -> Connection # is_valid=False on the shared row → cross-domain (C7, B3) + list_connections(*, project_id, ...) -> list[Connection] # backs /tools|/triggers/connections views + usage(*, project_id, connection_id) -> Usage # "used by tools / N subs" (what C7 ships) + +# ADAPTER PORT — provider-keyed, returns provider data. The 4 Composio auth verbs move behind THIS. +ConnectionsGatewayInterface: + initiate_connection(*, request: ConnectionRequest) -> ConnectionResponse + get_connection_status(*, provider_connection_id) -> dict + refresh_connection(*, provider_connection_id, ...) -> dict + revoke_connection(*, provider_connection_id) -> bool + +Connection DTO: { id (ca_*), project_id, provider, integration, slug, status, ... } +gateway_connections columns: (unchanged from tool_connections; already domain-neutral) +``` + +`ToolsService` delegates connection management to `ConnectionsService`. `ToolsGatewayInterface` +keeps only the tool-specific verbs (`execute`, catalog); the connection auth verbs move out to +`ConnectionsGatewayInterface` (implemented by a shared `ComposioConnectionsAdapter`). + +## Functional deps + +None — root. + +## Stubs needed + +None. + +## Decisions (RESOLVED — locked by orchestrator) + +- **B1** = Option A: full extract, two names (`ConnectionsService` + `ConnectionsGatewayInterface`), + no `connections → tools` import. WS3/WS5 freeze against `ConnectionsService`. +- **B2** = do not rename the `query_tool_connections` operation_id; only forced table refs change. +- **B3 / C7** = local-only `is_valid=False` revoke, cross-domain via the shared row; ship the + `usage()` read. Subscription delete must not revoke the connection. + +## Acceptance criteria + +- Every existing `/tools/connections` test passes **unchanged** (contract-frozen invariant). +- Migration up/down clean on **both** editions; `core_oss` chain head advances; legacy `core` + untouched. +- connect / refresh / revoke still work end-to-end via `/tools/connections`. +- (No triggers-side AC — no consumer yet.) + +## Risk + +The only PR that edits shipped tools code. Keep it a pure refactor + rename — **no behavior +change visible at `/tools`**. Largest blast radius; reviewed and merged first (it is WL0). diff --git a/docs/designs/gateway-triggers/wp/WP0-status.md b/docs/designs/gateway-triggers/wp/WP0-status.md new file mode 100644 index 0000000000..e91b90318c --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP0-status.md @@ -0,0 +1,65 @@ +# WP0 — Status + +**Lane** WL0 · **Stream** WS0 · **Branch** `wp0-connections-extract` (not yet created) + +| Field | Value | +|-------|-------| +| State | IMPLEMENTED (awaiting orchestrator commit/PR) — B1/B2/B3 resolved in spec | +| Contract frozen (WS-PRE) | ☑ `ConnectionsService` + `ConnectionsGatewayInterface` + `Connection`/`Usage` DTOs | +| Branch created | ☐ (orchestrator) | +| Subagent | WP0 impl | +| PR | — (orchestrator) | + +## Checklist + +- [x] Migration: `tool_connections` → `gateway_connections` in `core_oss` (both editions) +- [x] `core/connections/` service + DAO interface + `ConnectionsService` + `ConnectionsGatewayInterface` +- [x] `dbs/postgres/connections/` DBE + DAO + mappings +- [x] Move Composio auth verbs into shared `ComposioConnectionsAdapter` +- [x] Repoint `ToolsService` + `/tools/connections` + `/callback` handlers (delegate) +- [x] C7 cross-domain revoke rule (local-only `is_valid=False`) + `usage()` read +- [x] AC: existing `/tools/connections` contract unchanged (14 operation_ids preserved, incl. `query_tool_connections`) +- [ ] AC: migration up/down clean, both editions (needs live DB — run in CI/stack) +- [ ] PR opened `--base main` (orchestrator) + +## Decisions + +- [x] C7 revoke rule confirmed: local-only `is_valid=False` on the shared row; no provider + call, no cascade; provider revoke stays on DELETE. `usage()` is read-only seam. + +## Notes + +All three prior blockers resolved per the updated spec and implemented: + +- **B1 (Option A, full extract):** `ConnectionsService` (project-scoped, owns + `gateway_connections`, returns `Connection` DTOs) is the WS3/WS5 contract; + `ConnectionsGatewayInterface` is the provider-keyed adapter port holding only the four + auth verbs, implemented by `ComposioConnectionsAdapter`. `ConnectionsDAOInterface` is + the persistence port. Nothing in `connections` imports from `tools`; `ToolsService` + depends on `ConnectionsService` (one-way). `ToolsGatewayInterface` keeps only catalog + + `execute`; `ComposioToolsAdapter` lost the four auth verbs and its `_delete` helper. +- **B2:** `operation_id="query_tool_connections"` left untouched. The table rename moved the + table-defining code wholesale into `dbs/postgres/connections/dbes.py` as + `gateway_connections` with `uq_/ix_gateway_connections_*`; the old `dbs/postgres/tools` + package (DBE/DAO/mappings) was deleted (full extract ⇒ no in-place patch and no duplicate + SQLAlchemy mapping of the same table). The `uq_` IntegrityError match moved with it. +- **B3 / C7:** `ConnectionsService.revoke_connection` keeps today's local-only semantics + verbatim (`is_valid=False`, no provider call, no cascade). `usage()` reports + `tools=True` / `subscriptions=0` (seam; no subscription consumer exists yet). + +Layout chosen `core/connections/` + `dbs/postgres/connections/` (flat, matching existing +`core/tools/` and `core/triggers/`), not a `gateway/` subtree — the task brief specified +the flat paths and no `gateway/` tree exists in the working copy. + +Migration authored once at `core_oss` head `oss000000002` (revises `oss000000001`), +rename-only via `op.rename_table` + `RENAME CONSTRAINT` + `RENAME INDEX`, with a clean +inverse `downgrade`. Legacy `core` chain (parked `e5f6a1b2c3d4`) untouched; `core_ee` not +touched. OAuth state utils moved to `core/connections/utils.py`; the callback URL still +points at `/tools/connections/callback` (handler stays on the tools router) so the public +contract is byte-for-byte unchanged. + +Acceptance tests added in both editions: +`oss/tests/pytest/acceptance/tools/test_tools_connections.py` and +`ee/tests/pytest/acceptance/tools/test_tools_connections.py` (DB-only query + 404 always +run; create/revoke gated on `COMPOSIO_API_KEY`). Updated the lifecycle-conventions unit +test to register `connections.dbes` instead of the deleted `tools.dbes`. diff --git a/docs/designs/gateway-triggers/wp/WP1-specs.md b/docs/designs/gateway-triggers/wp/WP1-specs.md new file mode 100644 index 0000000000..c8e8afc5a7 --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP1-specs.md @@ -0,0 +1,66 @@ +# WP1 — Triggers skeleton + events catalog + adapter + +**Lane** WL1 (anchor WL0) · **Stream** WS1 (api) · **Area** api + +Parent docs: [`../plan.md`](../plan.md) §4, [`../gap.md`](../gap.md) §2.2, [`../mimics.md`](../mimics.md) (Triggers vs Tools, Part A). + +## Goal + +Stand up the triggers domain skeleton, the read-only **events** catalog, and the +`ComposioTriggersAdapter` that later WPs call to manage `ti_*` instances. + +## Closes (gap items) + +E1, E2, E3, E4 — and resolves **E5** (verify v3 REST paths). + +## Scope + +- **Skeleton** — `apis/fastapi/triggers/`, `core/triggers/`, `dbs/postgres/triggers/` + (mirror the tools layout; `action → event`). +- **Adapter** — `ComposioTriggersAdapter` (own httpx client, no SDK; `_get/_post/_delete` + + slug mapping modeled on `ComposioToolsAdapter`) behind `TriggersGatewayInterface`: + `list_events`, `get_event`, `create_subscription`, `set_subscription_status`, + `delete_subscription`. +- **Catalog** — `/triggers/catalog/.../integrations/{i}/events/{event_key}` returning the + event's `trigger_config` JSON Schema (analogue of an action's `input_parameters`). +- **Wiring** — `triggers` block in `entrypoints/routers.py` next to tools; adapter built + only when `env.composio.enabled`. +- **Permission** — introduce a dedicated **`VIEW_TRIGGERS`** permission (mirror the tools + triad in `api/ee/src/core/access/permissions/types.py`: add a `# Triggers` block and + register `VIEW_TRIGGERS` into the viewer `default_permissions`). Catalog routes gate on + `Permission.VIEW_TRIGGERS` — **do NOT reuse `VIEW_TOOLS`**. +- **E5** — verify exact Composio v3 REST paths (`triggers_types`, `trigger_instances/...`) + against the live OpenAPI spec; SDK method names are stable, paths must be confirmed. + +## Contracts this WP freezes (consumed by WS3, WS5 — freeze in WS-PRE) + +```text +TriggersGatewayInterface: + list_events(*, provider, integration) -> list[Event] + get_event(*, event_key) -> EventType # carries trigger_config JSON Schema + create_subscription(*, project_id, event_key, connected_account_id, trigger_config) -> "ti_*" + set_subscription_status(*, trigger_id, enabled: bool) -> None + delete_subscription(*, trigger_id) -> None +Catalog HTTP: GET /triggers/catalog/providers/{p}/integrations/{i}/events[/{event_key}] +Event DTO: { key, provider, integration, trigger_config: <JSONSchema>, ... } +``` + +## Functional deps + +None in-feature (uses `env.composio`, not the connection). Root in the §1 DAG. + +## Stubs needed + +None. + +## Decision to lock first + +**E5 — exact v3 REST paths** (verify vs live OpenAPI; the adapter can't be written +correctly without them). + +## Acceptance criteria (both editions) + +- Browse providers / integrations / events. +- Fetch one event's `trigger_config` schema. +- Catalog empty / disabled when `env.composio` unset. +- (Real adapter calls need live Composio creds — gate the integration test on that.) diff --git a/docs/designs/gateway-triggers/wp/WP1-status.md b/docs/designs/gateway-triggers/wp/WP1-status.md new file mode 100644 index 0000000000..a1d9102275 --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP1-status.md @@ -0,0 +1,43 @@ +# WP1 — Status + +**Lane** WL1 · **Stream** WS1 · **Branch** `wp1-events-catalog` (not yet created) + +| Field | Value | +|-------|-------| +| State | CODE COMPLETE (awaiting orchestrator commit/PR) | +| Contract frozen (WS-PRE) | ☑ `TriggersGatewayInterface` + `Event` DTO + catalog routes (implemented as written) | +| Branch created | ☐ (anchor `wp0-connections-extract`) | +| Subagent | — | +| PR | — | + +## Checklist + +- [x] Domain skeleton (apis/fastapi/triggers, core/triggers, dbs/postgres/triggers) +- [x] `ComposioTriggersAdapter` behind `TriggersGatewayInterface` (catalog + subscription verbs) +- [x] Events catalog routes + `trigger_config` schema return +- [x] Wiring in `entrypoints/routers.py` (gated on `env.composio.enabled`; lifespan close added) +- [x] E5: v3 REST paths verified vs live Composio API reference +- [x] AC: browse + fetch schema, both editions (provider catalog ungated; event browse gated on COMPOSIO_API_KEY) +- [ ] PR opened `--base wp0-connections-extract` (orchestrator) + +## Decisions + +- [x] E5 paths confirmed (verified against live Composio API reference, docs.composio.dev): + - List trigger types: `GET /triggers_types` (query `toolkit_slugs`, `limit`, `cursor`) + - Get one trigger type (config schema): `GET /triggers_types/{slug}` + - Create/upsert instance: `POST /trigger_instances/{slug}/upsert` (body `connected_account_id`, `trigger_config`) + - Enable/disable instance: `PATCH /trigger_instances/manage/{trigger_id}` (body `status` = `"enable"`/`"disable"`) + - Delete instance: `DELETE /trigger_instances/manage/{trigger_id}` + - All paths are relative to `env.composio.api_url` (default `/api/v3`); adapter builds `f"{api_url}{path}"` exactly like `ComposioToolsAdapter`. Docs currently surface these under the `v3.1` minor; the path *segments* (what E5 asked to confirm) are stable across v3/v3.1 and we keep the shared `env.composio.api_url` base. + +## Notes / blockers + +- E5 resolved without live creds: paths confirmed from the public Composio API reference (no auth needed). +- WP1 adds **no new env var**: it reuses the existing `env.composio` (enabled = key present). + `COMPOSIO_WEBHOOK_SECRET` is deliberately deferred to WP4 (ingress, gap I2) — adding it + now would be a consumer-less dead config. +- `dbs/postgres/triggers/` is an empty package skeleton in WP1 — the `subscriptions`/`deliveries` + tables + DAO + mappings are WP3 scope, so no DBE/migration here. +- EE catalog is gated on the existing `VIEW_TOOLS` permission (no `VIEW_TRIGGERS` introduced — + triggers share the gateway permission surface, per gap non-goal "no EE-only gating beyond tools"). +- Files changed listed in the final report to the orchestrator. diff --git a/docs/designs/gateway-triggers/wp/WP2-specs.md b/docs/designs/gateway-triggers/wp/WP2-specs.md new file mode 100644 index 0000000000..1b4fb11961 --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP2-specs.md @@ -0,0 +1,51 @@ +# WP2 — Resolver promotion (SDK + webhooks) + +**Lane** WL2 (anchor WL1) · **Stream** WS2 (sdk+webhooks) · **Area** sdk + webhooks + +Parent docs: [`../plan.md`](../plan.md) §4, [`../gap.md`](../gap.md) §2.5 (M1), [`../mapping.md`](../mapping.md) §5/§6. + +## Goal + +Promote the mapping resolver to the SDK under a neutral name so triggers and webhooks both +consume it without a cross-domain import. A complete, testable change on its own — its **live +consumer today is webhooks**, independent of triggers entirely. + +## Closes (gap items) + +M1. + +## Scope + +- Move `resolve_payload_fields` (`core/webhooks/delivery.py:95`) to + `agenta.sdk.utils.resolvers`, renamed **`resolve_target_fields`** (next to the existing + `resolve_json_selector` at `:114`). +- Update the webhooks call site to the new name/location. +- Pure move + rename — **no behavior change**. (It resolves a template into *a* target — + whole body for webhooks, `data.inputs` for triggers — hence the neutral name.) + +## Contracts this WP freezes (consumed by WS4 — freeze in WS-PRE) + +```text +agenta.sdk.utils.resolvers.resolve_target_fields(template, context) -> dict + # template: arbitrary JSON; leaves with $/ selectors resolved against context, else literal + # context: { event, subscription, scope } (allowlisted slots) + # null-on-miss, depth-capped (MAX_RESOLVE_DEPTH); default template "$" = whole context +``` + +## Functional deps + +None in-feature. Root in the §1 DAG. + +## Stubs needed + +None. + +## Decision to lock first + +None hard. (Confirm the SDK module path `agenta.sdk.utils.resolvers` is where it lands.) + +## Acceptance criteria + +- Existing **webhook delivery tests pass unchanged** against the renamed/relocated resolver. +- `resolve_target_fields` importable from `agenta.sdk.utils.resolvers`; no triggers→webhooks + import path introduced. diff --git a/docs/designs/gateway-triggers/wp/WP2-status.md b/docs/designs/gateway-triggers/wp/WP2-status.md new file mode 100644 index 0000000000..55d03ceebe --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP2-status.md @@ -0,0 +1,43 @@ +# WP2 — Status + +**Lane** WL2 · **Stream** WS2 · **Branch** `wp2-resolver-promote` (not yet created) + +| Field | Value | +|-------|-------| +| State | IMPLEMENTED (awaiting commit by orchestrator) | +| Contract frozen (WS-PRE) | ☑ `resolve_target_fields(template, context)` signature | +| Branch created | ☐ (anchor `wp1-events-catalog`) | +| Subagent | WP2 build agent | +| PR | — | + +## Checklist + +- [x] Move `resolve_payload_fields` → `agenta.sdk.utils.resolvers.resolve_target_fields` +- [x] Update webhooks call site +- [x] AC: webhooks delivery suite green, unchanged +- [ ] PR opened `--base wp1-events-catalog` + +## Decisions + +- [x] SDK module path confirmed — `sdks/python/agenta/sdk/utils/resolvers.py` + already exists and exports `resolve_json_selector`; `resolve_target_fields` + added next to it. No conflict. + +## Notes / blockers + +- Pure move + rename, no behavior change. `MAX_RESOLVE_DEPTH` (=10) moved with the + function into the SDK resolvers module (it only governed this recursion). +- Webhooks `delivery.py` now imports `resolve_target_fields` from the SDK and dropped + its local `resolve_payload_fields` + `MAX_RESOLVE_DEPTH`. +- Test file `test_webhooks_tasks.py`: imports + the `resolve_json_selector` patch target + repointed to `agenta.sdk.utils.resolvers`; assertions unchanged. All 19 tests pass. +- No triggers code touched; no triggers→webhooks import path introduced. +- Env note: the locally installed editable `agenta` resolves to the sibling `vibes` + worktree, so tests were run with `PYTHONPATH=.../application/sdks/python` to exercise + the edited SDK in this tree. + +## Files changed (for the orchestrator) + +- `sdks/python/agenta/sdk/utils/resolvers.py` (add `resolve_target_fields` + `MAX_RESOLVE_DEPTH`) +- `api/oss/src/core/webhooks/delivery.py` (import + call site; drop local fn/const) +- `api/oss/tests/pytest/unit/webhooks/test_webhooks_tasks.py` (import + patch target rename) diff --git a/docs/designs/gateway-triggers/wp/WP3-specs.md b/docs/designs/gateway-triggers/wp/WP3-specs.md new file mode 100644 index 0000000000..e92a38c16d --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP3-specs.md @@ -0,0 +1,64 @@ +# WP3 — Subscriptions + deliveries + +**Lane** WL3 (anchor WL2) · **Stream** WS3 (api) · **Area** api + +Parent docs: [`../plan.md`](../plan.md) §4, [`../gap.md`](../gap.md) §2.3, [`../mimics.md`](../mimics.md) (Triggers vs Webhooks), +[`../mapping.md`](../mapping.md) §3–§4. + +## Goal + +The two-table heart of the domain, modeled on webhooks' `webhook_subscriptions` + +`webhook_deliveries`. Functional as **subscription CRUD** before any dispatch exists. + +## Closes (gap items) + +S1, S2, S3, S4, S5. + +## Scope + +- **`subscriptions` table** (FlagsDBA enabled/valid, DataDBA): `ti_*`, `trigger_config`, + `inputs_fields` (the mapping template), destination `references`/`selector`, the bound + **workflow ref**, **FK → `gateway_connections`**. Many per connection. +- **`deliveries` table** (modeled on `webhook_deliveries`): resolved `inputs`, workflow + `references`, `result`/`error`, plus the `metadata.id` **dedup column** (I4). +- **DBA mixins** for both (mirror `dbs/postgres/webhooks/dbas.py`; tools has none). +- **Migration** authored once in the shared `core_oss` chain (both editions, per WP0's rule). +- **Subscription CRUD** `/triggers/subscriptions/` · `/query` · `/{id}` · `/{id}/refresh` · + `/{id}/revoke` — create/disable/delete the Composio `ti_*` through the adapter + (`TriggersGatewayInterface.create_subscription` etc.), referencing a shared connection. + Deleting a subscription must **not** revoke the connection (C7). +- **Delivery read** routes `/triggers/deliveries` · `/{id}` · `/query`. + +## Contracts this WP freezes (consumed by WS4, WS6 — freeze in WS-PRE) + +```text +Subscription DTO: { id, project_id, connection_id (FK), event_key, ti_id, trigger_config, + inputs_fields, references, selector, enabled, valid, ... } +Delivery DTO: { id, subscription_id, event_id (metadata.id), inputs, references, result, error, ... } +HTTP: /triggers/subscriptions/{,query,{id},{id}/refresh,{id}/revoke}; /triggers/deliveries/{,{id},query} +DAO surface (for WP4): get_subscription_by_trigger_id, write_delivery, dedup_seen(event_id) +``` + +## Functional deps (fan-in) + +- **WP0** — `subscriptions` FKs `gateway_connections`. +- **WP1** — `create_subscription` builds the `ti_*` via `TriggersGatewayInterface` (the + adapter, **not** the catalog routes). + +## Stubs needed (until deps merge) + +- `ConnectionsGatewayInterface` (WP0) — stub the connection lookup/FK target. +- `TriggersGatewayInterface` (WP1) — stub `create_subscription`/`set_status`/`delete`. + +Both against their frozen WS-PRE contracts; mock in unit tests. + +## Decisions to lock first + +- **Idempotency store (I4)** — lean: a `metadata.id` dedup column on `deliveries`. +- **Default mapping + validation posture (M8)** — inputs-only default; schema validation a stretch. + +## Acceptance criteria (both editions) + +- Create a subscription on a shared connection bound to a workflow. +- List / disable / delete it; deleting it leaves the connection intact (C7). +- Deliveries list returns rows. diff --git a/docs/designs/gateway-triggers/wp/WP3-status.md b/docs/designs/gateway-triggers/wp/WP3-status.md new file mode 100644 index 0000000000..b431318d6a --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP3-status.md @@ -0,0 +1,33 @@ +# WP3 — Status + +**Lane** WL3 · **Stream** WS3 · **Branch** `wp3-subscriptions` (not yet created) + +| Field | Value | +|-------|-------| +| State | NOT STARTED | +| Contract frozen (WS-PRE) | ☐ Subscription/Delivery DTOs + routes + DAO surface | +| Consumes frozen | ☐ ConnectionsGW (WP0) ☐ TriggersGW (WP1) | +| Branch created | ☐ (anchor `wp2-resolver-promote`) | +| Subagent | — | +| PR | — | + +## Checklist + +- [ ] `subscriptions` table (FlagsDBA, DataDBA, FK → gateway_connections) +- [ ] `deliveries` table (+ metadata.id dedup column) +- [ ] DBA mixins (mirror webhooks/dbas.py) +- [ ] Migration in `core_oss` (both editions) +- [ ] Subscription CRUD routes + adapter calls (ti_* lifecycle) +- [ ] Delivery read routes +- [ ] Stub ConnectionsGW (WP0) + TriggersGW (WP1) until merged +- [ ] AC: create/list/disable/delete; delete leaves connection intact +- [ ] PR opened `--base wp2-resolver-promote` + +## Decisions + +- [ ] I4 idempotency store (dedup column) +- [ ] M8 default mapping + validation posture + +## Notes / blockers + +_(none yet)_ diff --git a/docs/designs/gateway-triggers/wp/WP4-specs.md b/docs/designs/gateway-triggers/wp/WP4-specs.md new file mode 100644 index 0000000000..71596abc24 --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP4-specs.md @@ -0,0 +1,58 @@ +# WP4 — Ingress + dispatch + +**Lane** WL4 (anchor WL3) · **Stream** WS4 (api) · **Area** api + +Parent docs: [`../plan.md`](../plan.md) §4, [`../gap.md`](../gap.md) §2.4 + §2.5, [`../mimics.md`](../mimics.md) (Triggers vs Billing; +Triggers vs Everything), [`../mapping.md`](../mapping.md) §3–§4. + +## Goal + +Close the loop in **one** functional unit: an inbound event is received, verified, scoped, +resolved, and acted on. Ingress lives here (not its own lane) because a verify-and-park +endpoint isn't functional — the receive path only becomes real once it dispatches. + +## Closes (gap items) + +I1, I2, I3, I4, I5, I6, M2, M3, M4, M5, M6, M7, M9 — and consumes **M1** (the resolver). + +## Scope — ingress half (mimic billing `/stripe/events/`) + +- `POST /triggers/composio/events/` — read raw body **before** parsing. +- HMAC-SHA256 verify over `{id}.{ts}.{body}` with `COMPOSIO_WEBHOOK_SECRET`; 401 bad sig; + 200 no-op when secret unset; add `COMPOSIO_WEBHOOK_SECRET` to `env`. +- Recover `project_id` from `metadata.user_id`; route `metadata.trigger_id` → local + subscription; 200-skip unknown/disabled; optional `target`-style env fan-out guard (I5). +- One-time project webhook-URL registration with Composio (I6). + +## Scope — dispatch half + +- Resolve `inputs_fields` via `resolve_target_fields` against `{event, subscription, scope}` + with `TRIGGER_EVENT_FIELDS` (M2, M3) into `data.inputs` **only**. +- Build the `WorkflowServiceRequest`: destination from the stored workflow `references`/ + `selector` (M4); call `WorkflowsService.invoke_workflow(project_id, user_id, request)` (M5). +- **System-initiated identity** (M6) — run as a resolved project-system `user_id`. +- **Async dispatch** (M7) — ack-fast + enqueue; ingress returns 2xx promptly. +- Real `metadata.id` dedup against `deliveries` (I4); write a delivery row per event with + outcome; dispatch retry policy (M9). + +## Functional deps (fan-in) + +- **WP3** — reads the subscription, writes a `deliveries` row (DTO + DAO surface). +- **WP2** — imports `resolve_target_fields`. + +## Stubs needed (until deps merge) + +- Subscription DTO/DAO (WP3) — stub `get_subscription_by_trigger_id` + `write_delivery`. +- `resolve_target_fields` (WP2) — import against the frozen signature. + +## Decisions to lock first + +Webhook-URL registration (I6), sync-vs-async (M7), system `user_id` (M6), retry policy (M9). + +## Acceptance criteria (both editions) + +- Forged signature → 401; unset secret → 200 no-op. +- Signed event for a known subscription → bound workflow invoked with the mapped inputs. +- Duplicate `metadata.id` → **single** invocation. +- Bad mapping / missing workflow → a `deliveries` **error row** (no workflow trace), still + 2xx to the provider. diff --git a/docs/designs/gateway-triggers/wp/WP4-status.md b/docs/designs/gateway-triggers/wp/WP4-status.md new file mode 100644 index 0000000000..2da5f306e1 --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP4-status.md @@ -0,0 +1,36 @@ +# WP4 — Status + +**Lane** WL4 · **Stream** WS4 · **Branch** `wp4-ingress-dispatch` (not yet created) + +| Field | Value | +|-------|-------| +| State | NOT STARTED | +| Consumes frozen | ☐ Subscription DTO/DAO (WP3) ☐ `resolve_target_fields` (WP2) | +| Branch created | ☐ (anchor `wp3-subscriptions`) | +| Subagent | — | +| PR | — | + +## Checklist + +- [ ] `POST /triggers/composio/events/` raw-body + HMAC verify + `COMPOSIO_WEBHOOK_SECRET` +- [ ] project/trigger scoping + 200-skip + target guard (I5) +- [ ] webhook-URL registration (I6) +- [ ] resolve `inputs_fields` → `data.inputs` (M2, M3) +- [ ] build request refs/selector (M4) + `invoke_workflow` (M5) +- [ ] system `user_id` (M6) +- [ ] async dispatch (M7) +- [ ] metadata.id dedup (I4) + delivery rows + retry (M9) +- [ ] Stub WP3 DAO + WP2 resolver until merged +- [ ] AC: 401 / no-op / invoke / dedup / error-row +- [ ] PR opened `--base wp3-subscriptions` + +## Decisions + +- [ ] I6 webhook-URL registration +- [ ] M7 sync vs async +- [ ] M6 system identity +- [ ] M9 retry policy + +## Notes / blockers + +_(none yet)_ diff --git a/docs/designs/gateway-triggers/wp/WP5-specs.md b/docs/designs/gateway-triggers/wp/WP5-specs.md new file mode 100644 index 0000000000..73d1e2a4cb --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP5-specs.md @@ -0,0 +1,43 @@ +# WP5 — Web: catalog + connections UI + +**Lane** WL5 (anchor WL1) · **Stream** WS5 (web) · **Area** web + +Parent docs: [`../plan.md`](../plan.md) §4, [`../gap.md`](../gap.md) §2.6 (F1 browse, F2). + +## Goal + +The browse half of the FE: providers / integrations / events and the connection list, on a +"Triggers" surface of a connected integration. + +## Closes (gap items) + +F1 (catalog/connect part), F2. + +## Scope + +- "Triggers" entry on a connected integration — browse events and their `trigger_config` + schema (WP1 catalog API). +- Show connections via `/triggers/connections`. +- Handle the **overlapping connection reads** across `/tools/connections` and + `/triggers/connections` (same shared rows, F2) — the FE must tolerate the same connection + appearing in both lists. +- Reuse the existing tools UI surfaces: `web/packages/agenta-entities/src/gatewayTool`, + `web/packages/agenta-entity-ui/src/gatewayTool`, `web/oss/src/components/pages/settings/Tools`. + +## Functional deps (fan-in) + +- **WP1** — the catalog API. +- **WP0** — the `/…/connections` view over `gateway_connections`. + +## Stubs needed (until deps merge) + +- Mock the catalog (WP1) and `/…/connections` (WP0) HTTP against their frozen shapes. + +## Decisions to lock first + +None hard (consumes frozen API shapes). + +## Acceptance criteria + +- Browse a connected integration's events. +- The same connection appears under **both** tools and triggers without a second connect. diff --git a/docs/designs/gateway-triggers/wp/WP5-status.md b/docs/designs/gateway-triggers/wp/WP5-status.md new file mode 100644 index 0000000000..eca1b9a1a4 --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP5-status.md @@ -0,0 +1,25 @@ +# WP5 — Status + +**Lane** WL5 · **Stream** WS5 · **Branch** `wp5-web-catalog` (not yet created) + +| Field | Value | +|-------|-------| +| State | NOT STARTED | +| Consumes frozen | ☐ catalog API (WP1) ☐ /…/connections (WP0) | +| Branch created | ☐ (anchor `wp1-events-catalog`) | +| Subagent | — | +| PR | — | + +## Checklist + +- [ ] "Triggers" surface on a connected integration +- [ ] Events browse + `trigger_config` schema view (WP1 API) +- [ ] Connections list via `/triggers/connections` +- [ ] F2: tolerate overlapping connection reads (tools ∩ triggers) +- [ ] Mock WP1/WP0 HTTP until merged +- [ ] AC: browse events; connection shows under both +- [ ] PR opened `--base wp1-events-catalog` + +## Notes / blockers + +_(none yet)_ diff --git a/docs/designs/gateway-triggers/wp/WP6-specs.md b/docs/designs/gateway-triggers/wp/WP6-specs.md new file mode 100644 index 0000000000..6370bc4da8 --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP6-specs.md @@ -0,0 +1,38 @@ +# WP6 — Web: subscriptions + deliveries UI + +**Lane** WL6 (anchor WL3) · **Stream** WS6 (web) · **Area** web + +Parent docs: [`../plan.md`](../plan.md) §4, [`../gap.md`](../gap.md) §2.6 (F1 subscribe, F3). + +## Goal + +The management half of the FE: create / manage subscriptions and view deliveries. + +## Closes (gap items) + +F1 (subscribe part), F3. + +## Scope + +- Create a subscription — pick event + bind workflow + author the mapping (`inputs_fields`) — + via the WP3 subscription API. +- List / disable / delete subscriptions. +- Deliveries audit view (`/triggers/deliveries`, F3 — deferrable past v1). + +## Functional deps + +- **WP3** only — the `/triggers/subscriptions` + `/triggers/deliveries` API. Independent of + WP4 (the management UI doesn't need dispatch to exist). + +## Stubs needed (until deps merge) + +- Mock the WP3 HTTP surface against its frozen shape. + +## Decisions to lock first + +None hard (consumes the frozen WP3 API). + +## Acceptance criteria + +- Create a workflow-bound subscription; list / disable / delete it. +- Deliveries view renders (empty until WP4 dispatch lands). diff --git a/docs/designs/gateway-triggers/wp/WP6-status.md b/docs/designs/gateway-triggers/wp/WP6-status.md new file mode 100644 index 0000000000..d96d03b64c --- /dev/null +++ b/docs/designs/gateway-triggers/wp/WP6-status.md @@ -0,0 +1,24 @@ +# WP6 — Status + +**Lane** WL6 · **Stream** WS6 · **Branch** `wp6-web-subscriptions` (not yet created) + +| Field | Value | +|-------|-------| +| State | NOT STARTED | +| Consumes frozen | ☐ /triggers/subscriptions + /deliveries (WP3) | +| Branch created | ☐ (anchor `wp3-subscriptions`) | +| Subagent | — | +| PR | — | + +## Checklist + +- [ ] Create subscription (event + workflow binding + mapping) +- [ ] List / disable / delete +- [ ] Deliveries audit view (F3, deferrable) +- [ ] Mock WP3 HTTP until merged +- [ ] AC: create/manage; deliveries renders empty +- [ ] PR opened `--base wp3-subscriptions` + +## Notes / blockers + +_(none yet)_ From 2936e889bfcb5e3071d6f16d46575a82841f375a Mon Sep 17 00:00:00 2001 From: Juan Pablo Vega <jp@agenta.ai> Date: Thu, 18 Jun 2026 17:57:07 +0200 Subject: [PATCH 2/4] docs(agents): clarify GitButler stacked-lane committing and silent push Document the staging model that bit us: changes assign to the stack (not a branch), so commit one lane at a time and verify each commit. Note that but rub by path goes stale after a mutation (use cliId), how to split one file across stacked lanes, branch-ref vs workspace-applied divergence, and that but push prints nothing on success (verify SHAs against the remote). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --- AGENTS.md | 35 ++++++++++++++++++++++++++++++++--- 1 file changed, 32 insertions(+), 3 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 14a73c0f3e..2bdc18f650 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -40,11 +40,40 @@ If so, use the `but` CLI instead of raw `git branch`/`git commit`: - `but pr new` needs interactive forge auth; use `but push <branch>` then `gh pr create --head <branch> --base <parent-or-main>` instead. For stacked PRs, set `--base` to the parent branch so each PR shows only its own diff. +- **`but push` prints NOTHING on success.** It is not a confirmation — always verify + the push landed by comparing SHAs: + `git ls-remote --heads origin <branch>` vs `git rev-parse <branch>`. They must match. - To update an already-committed file, `but absorb <path>` amends it into the right commit; force-push with `but push <branch> -f`. -- To commit to a specific branch in a stack, stage the files to it first - (`but rub <path> <branch>`), then `but commit <branch> --only`. `but commit` - alone sweeps ALL uncommitted changes into that branch. + +### Committing to specific lanes in a stack (the part that bites) + +Changes are assigned to the **stack**, not to an individual branch. `but rub <file> +<branch>` and `but commit <branch> --only` both operate on the stack's *assigned-changes* +set — `--only` commits **whatever is currently assigned** to the named branch, regardless +of which branch name you used when staging. So: + +- **Never pre-stage multiple lanes' files and then commit them one lane at a time.** The + first `but commit --only` sweeps the entire assigned set into that one branch (the others + end up empty or scrambled). Instead, work **one lane at a time**: assign exactly that + lane's files → `but commit <branch> --only` → **verify** → then assign the next lane's + files. Keep the assigned set equal to exactly one lane's files at each commit. +- **Verify every commit immediately:** `git show --stat --name-only <branch>`. If a file + from another lane leaked in, stop and fix before continuing. +- **`but rub` by path goes stale after any mutation.** Every `but` mutation kicks a + background sync that invalidates the path index, so the *next* path-based + `but rub <path> ...` often fails with "Source '<path>' not found". Use the stable + **cliId** instead (the 2-4 char code in `but status` / `but status --json`): + `but rub <cliId> <target>`. cliIds survive across the sync; paths don't. +- **Splitting one file across two stacked lanes** (e.g. `routers.py` where the lower lane + owns half the edit and the upper lane the other half): you cannot split mixed hunks + reliably. Instead use sequential working-tree states — make the file the lower lane's + version, commit it to the lower lane; then edit the file to add the upper lane's delta + and `but rub <fileCliId> <upperCommitCliId>` to amend that delta into the upper commit. +- The **branch ref can diverge from the workspace-applied commit** mid-session (after + absorb/amend/rebase). The **working tree is the source of truth**; `but push` pushes the + applied state. Don't panic if `git diff <branch> -- <file>` shows a delta while + `git status` is clean — verify against `git show "<branch>:<file>"` and re-push. ### Hard-won gotchas (don't relearn these) From 854611200c308f8519b8af7009d5bab0bf9f992b Mon Sep 17 00:00:00 2001 From: Juan Pablo Vega <jp@agenta.ai> Date: Thu, 18 Jun 2026 18:26:23 +0200 Subject: [PATCH 3/4] docs(triggers): update WP3 and WP5 status after implementation Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --- .../designs/gateway-triggers/wp/WP3-status.md | 69 ++++++++++++------ .../designs/gateway-triggers/wp/WP5-status.md | 71 ++++++++++++++++--- 2 files changed, 109 insertions(+), 31 deletions(-) diff --git a/docs/designs/gateway-triggers/wp/WP3-status.md b/docs/designs/gateway-triggers/wp/WP3-status.md index b431318d6a..16863c239c 100644 --- a/docs/designs/gateway-triggers/wp/WP3-status.md +++ b/docs/designs/gateway-triggers/wp/WP3-status.md @@ -1,33 +1,60 @@ # WP3 — Status -**Lane** WL3 · **Stream** WS3 · **Branch** `wp3-subscriptions` (not yet created) +**Lane** WL3 · **Stream** WS3 · **Branch** `wp3-subscriptions` (created by orchestrator) | Field | Value | |-------|-------| -| State | NOT STARTED | -| Contract frozen (WS-PRE) | ☐ Subscription/Delivery DTOs + routes + DAO surface | -| Consumes frozen | ☐ ConnectionsGW (WP0) ☐ TriggersGW (WP1) | -| Branch created | ☐ (anchor `wp2-resolver-promote`) | -| Subagent | — | +| State | IMPLEMENTED (pending commit + live-API test run) | +| Contract frozen (WS-PRE) | ☑ Subscription/Delivery DTOs + routes + DAO surface | +| Consumes frozen | ☑ ConnectionsGW (WP0) ☑ TriggersGW (WP1) | +| Branch created | (orchestrator) | +| Subagent | WP3 build | | PR | — | ## Checklist -- [ ] `subscriptions` table (FlagsDBA, DataDBA, FK → gateway_connections) -- [ ] `deliveries` table (+ metadata.id dedup column) -- [ ] DBA mixins (mirror webhooks/dbas.py) -- [ ] Migration in `core_oss` (both editions) -- [ ] Subscription CRUD routes + adapter calls (ti_* lifecycle) -- [ ] Delivery read routes -- [ ] Stub ConnectionsGW (WP0) + TriggersGW (WP1) until merged -- [ ] AC: create/list/disable/delete; delete leaves connection intact -- [ ] PR opened `--base wp2-resolver-promote` - -## Decisions - -- [ ] I4 idempotency store (dedup column) -- [ ] M8 default mapping + validation posture +- [x] `trigger_subscriptions` table (FlagsDBA enabled/valid, DataDBA, FK → gateway_connections) +- [x] `trigger_deliveries` table (+ `event_id` = provider `metadata.id` dedup column, unique per subscription) +- [x] DBA mixins (mirror webhooks/dbas.py) — `dbs/postgres/triggers/dbas.py` +- [x] Migration in `core_oss` (`oss000000003`, down_revision `oss000000002`; runs in both editions) +- [x] Subscription CRUD routes + adapter calls (ti_* create / set-status / delete) +- [x] Delivery read routes (`/triggers/deliveries`, `/{id}`, `/query`) +- [x] DAO surface for WP4: `get_subscription_by_trigger_id`, `write_delivery`, `dedup_seen` +- [x] AC tests (OSS + EE): list/query/404 DB-only; create/list/disable/delete + C7 gated on COMPOSIO_API_KEY +- [ ] PR opened `--base wp2-resolver-promote` (orchestrator) + +## Decisions (locked, built to) + +- [x] I4 idempotency — `event_id` (String) dedup column on `trigger_deliveries`, unique on + `(project_id, subscription_id, event_id)`; `write_delivery` upserts on it, `dedup_seen` checks it. +- [x] M8 default mapping — inputs-only; `inputs_fields` template stored on the subscription, resolved + (by WP4) via the promoted `agenta.sdk.utils.resolvers.resolve_target_fields`. No schema validation. + +## Implementation notes + +- Tables named `trigger_subscriptions` / `trigger_deliveries` (domain-prefixed, mirroring + `webhook_subscriptions`/`webhook_deliveries`) — NOT bare `subscriptions`/`deliveries`, which would + collide with EE billing subscriptions. +- Subscription DTO nests `event_key`/`ti_id`/`trigger_config`/`inputs_fields`/`references`/`selector` + under `data` (exactly as webhooks nests `event_types`/`payload_fields` under `data`); `connection_id`, + `enabled`, `valid` are top-level. The frozen field inventory is satisfied; nesting follows the + webhooks precedent it mirrors. +- `enabled`/`valid` persist in the FlagsDBA `flags` JSONB (`{"enabled":..,"valid":..}`). +- C7 enforced: `delete_subscription` / `revoke_subscription` only touch the provider trigger instance + (`ti_*`) via the adapter, never the shared `gateway_connections` row. +- EE permissions: added `EDIT_TRIGGERS` to EDITOR_PERMISSIONS and `RUN_TRIGGERS` to ANNOTATOR_PERMISSIONS + (parallel to `EDIT_TOOLS`/`RUN_TOOLS`) so the developer role can actually exercise subscription CRUD — + the enum values existed but were ungranted to every role except owner. See blocker note below. ## Notes / blockers -_(none yet)_ +- **Testing seam (not a blocker, but a constraint):** acceptance tests run over HTTP against a live API, + so the Composio adapter cannot be dependency-injected/mocked. The instruction "mock the adapter" is + satisfied in spirit by gating the adapter-dependent path (create → ti_* → disable → delete, plus the + C7 connection-intact assertion) on `COMPOSIO_API_KEY`, exactly as the existing tools/connections and + triggers/catalog suites do. DB-only reads/queries/404s run unconditionally and prove the migration + landed. If a true adapter mock is wanted, it needs a unit-test harness against `TriggersService` + (out of WP3's acceptance-test scope). +- **EE permission grant (flagged for review):** I added `EDIT_TRIGGERS`/`RUN_TRIGGERS` to the + editor/annotator role sets. This is the minimal change to make the locked `EDIT_TRIGGERS` gating + functional for non-owner roles; if WP1 intended a different role mapping, adjust there. diff --git a/docs/designs/gateway-triggers/wp/WP5-status.md b/docs/designs/gateway-triggers/wp/WP5-status.md index eca1b9a1a4..617c70644f 100644 --- a/docs/designs/gateway-triggers/wp/WP5-status.md +++ b/docs/designs/gateway-triggers/wp/WP5-status.md @@ -4,22 +4,73 @@ | Field | Value | |-------|-------| -| State | NOT STARTED | -| Consumes frozen | ☐ catalog API (WP1) ☐ /…/connections (WP0) | +| State | IMPLEMENTED (awaiting branch/PR) | +| Consumes frozen | ☑ catalog API (WP1) ☑ /…/connections (WP0) | | Branch created | ☐ (anchor `wp1-events-catalog`) | -| Subagent | — | +| Subagent | WS5 | | PR | — | ## Checklist -- [ ] "Triggers" surface on a connected integration -- [ ] Events browse + `trigger_config` schema view (WP1 API) -- [ ] Connections list via `/triggers/connections` -- [ ] F2: tolerate overlapping connection reads (tools ∩ triggers) -- [ ] Mock WP1/WP0 HTTP until merged -- [ ] AC: browse events; connection shows under both +- [x] "Triggers" surface on a connected integration (settings tab + section) +- [x] Events browse + `trigger_config` schema view (WP1 API) +- [x] Connections list via `/triggers/connections` +- [x] F2: tolerate overlapping connection reads (tools ∩ triggers) +- [x] Mock WP1/WP0 HTTP until merged (unit tests stub axios at the boundary) +- [x] AC: browse events; connection shows under both - [ ] PR opened `--base wp1-events-catalog` +## What was built + +New `@agenta/entities/gatewayTrigger` (state + queries) and +`@agenta/entity-ui/gatewayTrigger` (events drawer), mirroring `gatewayTool`. New OSS +`settings/Triggers` surface wired as a `triggers` settings tab (gated by `isToolsEnabled()`, +the shared Composio gate). The Triggers section lists the shared connections and opens an +events drawer per connection; selecting an event shows its `trigger_config` schema +(read-only, via the reused `SchemaForm`). + +### Files + +Entities (`web/packages/agenta-entities/`): + +- `src/gatewayTrigger/core/types.ts` (+ `core/index.ts`) +- `src/gatewayTrigger/api/{client,api,index}.ts` +- `src/gatewayTrigger/state/{atoms,index}.ts` +- `src/gatewayTrigger/hooks/{useCatalogEvents,useTriggerEvent,useTriggerConnections,index}.ts` +- `src/gatewayTrigger/index.ts` +- `tests/unit/gatewayTriggerApi.test.ts` +- `package.json` (added `./gatewayTrigger` export) + +Entity-UI (`web/packages/agenta-entity-ui/`): + +- `src/gatewayTrigger/drawers/TriggerEventsDrawer.tsx` +- `src/gatewayTrigger/index.ts` +- `package.json` (added `./gatewayTrigger` export) + +OSS (`web/oss/`): + +- `src/components/pages/settings/Triggers/Triggers.tsx` +- `src/components/pages/settings/Triggers/components/GatewayTriggersSection.tsx` +- `src/pages/w/[workspace_id]/p/[project_id]/settings/index.tsx` (triggers tab) +- `src/components/Sidebar/SettingsSidebar.tsx` (triggers menu item) + ## Notes / blockers -_(none yet)_ +- **Fern client gap (follow-up, not a blocker):** the shipped WP1 catalog API is NOT yet + in the Fern-generated `@agentaai/api-client` (no `triggers` resource). Per the WS5 stub + strategy this layer uses the shared axios instance with zod boundary validation (the + local schemas mirror `core/triggers/dtos.py` + `triggers/models.py` verbatim). When the + client is regenerated with a `triggers` resource, `gatewayTrigger/api/*` collapses onto + `getAgentaSdkClient().triggers` the same way `gatewayTool` does — a mechanical swap. +- **`/triggers/connections` consumed against the frozen WP0 shape, not yet shipped.** The + triggers router (`api/oss/src/apis/fastapi/triggers/router.py`) currently exposes only + the catalog routes; the `/triggers/connections` view over `gateway_connections` (WP0) is + not mounted there yet. The FE calls `POST /triggers/connections/query` mirroring + `POST /tools/connections/query` (same `{count, connections: Connection[]}` shape, same + shared rows). This is exactly the WP0 dep WS5 stubs until it merges; unit tests cover the + request/response shape. No backend change is in WP5 scope. +- **F2 handled explicitly:** trigger connections use their own React-Query keys + (`["triggers", "connections", …]`), distinct from tools (`["tools", …]`), so the same + shared row in both lists causes no cache or rowKey collision. The connection TS type is + aliased to the gatewayTool type so the two lists are byte-compatible; no duplicate-connect + path exists on the triggers surface (it only reads + browses events). From cbebabe704c7f00e9fe011ca1eb06a7cdd908be9 Mon Sep 17 00:00:00 2001 From: Juan Pablo Vega <jp@agenta.ai> Date: Thu, 18 Jun 2026 18:35:36 +0200 Subject: [PATCH 4/4] docs(agents): document GitButler linear-stack vs PR-base fan-out Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --- AGENTS.md | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) diff --git a/AGENTS.md b/AGENTS.md index 2bdc18f650..e04ab30fbb 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -75,6 +75,35 @@ of which branch name you used when staging. So: applied state. Don't panic if `git diff <branch> -- <file>` shows a delta while `git status` is clean — verify against `git show "<branch>:<file>"` and re-push. +### Stacks are linear; a fan-out is expressed through PR bases, not graph shape + +A GitButler **stack** is a linear series. `but branch new <name> --anchor <parent>` does NOT +create a sibling of `<parent>` — it **inserts the new branch into the line** on top of it. So +anchoring two branches on the same parent produces `parent → first → second`, not two children +of `parent`. `but branch new <name>` with **no** anchor makes a separate parallel stack, but a +parallel stack branches off the workspace base (main), so a branch that genuinely depends on an +ancestor's commits can't live there with a clean diff. + +This matters when a design's dependency tree fans out (e.g. a web lane and an SDK lane that both +depend on an API lane but not on each other). You cannot draw that fan-out in the git graph here. +You don't need to. The clean per-PR diff is a **PR-base** property, not a graph-shape property: +a stacked branch contains every commit below it, and GitHub shows only the delta against the base +you set. So put everything in **one linear stack in dependency order** and set each PR's base to +the branch directly below it. Order independent lanes however you like (sort by fewest conflicts); +lanes that touch disjoint files (e.g. `web/**` vs `api/**`) can sit anywhere in the line. + +- Build the line with `but move <branch> <target-branch>` (stacks `<branch>` on top of `<target>`) + and `but move <branch> zz` (tears `<branch>` off into its own parallel stack). Use these to + reorder after the fact; take a `but oplog snapshot` first. +- **Verify the line by diffing, not by eyeballing the tree.** For each branch, run + `git diff --name-only <base>..<branch>` where `<base>` is the branch below it. The file list + must be exactly that lane's files. If a lower lane's files appear, the order is wrong (a lane got + inserted into another's ancestry) — `but move` it out of the way and re-diff. +- A branch torn off to its own parallel stack (base = main) gives a **wrong** diff against an + ancestor branch: `git diff <ancestor>..<torn-off>` reverses the ancestor's own changes (their + merge base is main). That's the tell that the branch needs to be stacked, not parallel. +- Set PR bases to match: bottom lane `--base main`, every other lane `--base <branch-below-it>`. + ### Hard-won gotchas (don't relearn these) - **GitButler series need linear history.** A stack of branches connected by