Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
61 changes: 61 additions & 0 deletions src/destinations/bigquery.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
---
title: BigQuery destinations
---

For more information on destinations, see the [Destinations](/destinations) page.

When new data is read from a SaaS instance via a [Read Action](/read-actions) or [Subscribe Action](/subscribe-actions), Ampersand inserts the payload as rows into your Google BigQuery table.

## Prerequisites

Before setting up a BigQuery destination, ensure that you have:
- A Google Cloud project with BigQuery enabled
- A dataset and table to insert into (see [Data format and table setup](#data-format-and-table-setup))
- A service account with the `BigQuery Data Editor` role (or `bigquery.tables.updateData` permission) on that dataset, and a JSON key for it

## Data format and table setup

Ampersand delivers each read or subscribe message as a JSON object. The full payload schema is defined in [webhook.yaml](https://github.com/amp-labs/openapi/blob/main/webhook/webhook.yaml).

The BigQuery destination writes each message into two columns: `id` (a generated identifier) and `payload` (the full message JSON, written as a string). Create the table with exactly these columns:

```sql
CREATE TABLE `my-project.my_dataset.my_table` (
id STRING,
payload STRING
);
```

## Create a BigQuery destination

Go to the [Destinations page](https://dashboard.withampersand.com/projects/_/destinations) in the Ampersand Dashboard and create a new BigQuery destination.

You'll need to provide:

| Field | Type | Required | Example | Description |
|-------|------|----------|---------|-------------|
| **Destination name** | string | Yes | `ampersandBigQuery` | Alias to reference in your `amp.yaml` file |
| **Project ID** | string | Yes | `my-gcp-project` | Google Cloud project ID |
| **Dataset ID** | string | Yes | `analytics` | BigQuery dataset ID |
| **Table ID** | string | Yes | `ampersand_events` | BigQuery table ID |
| **Service account credentials** | string | Yes | | Service account key JSON with write access to the table |
| **Batch size** | integer | No | `1000` | Rows to buffer before flushing a batch (defaults to 1000) |
| **Max wait (seconds)** | integer | No | `30` | Max seconds to wait before flushing a batch (defaults to 30) |

## Refer to the destination in your integration

After creating your BigQuery destination, reference it in your `amp.yaml` file:

```yaml
specVersion: 1.0.0
integrations:
- name: salesforceToBigQuery
displayName: Salesforce to BigQuery
provider: salesforce
read:
objects:
- objectName: account
destination: ampersandBigQuery
- objectName: contact
destination: ampersandBigQuery
```
69 changes: 69 additions & 0 deletions src/destinations/clickhouse.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
---
title: ClickHouse destinations
---

For more information on destinations, see the [Destinations](/destinations) page.

When new data is read from a SaaS instance via a [Read Action](/read-actions) or [Subscribe Action](/subscribe-actions), Ampersand inserts the payload as rows into your ClickHouse table.

## Prerequisites

Before setting up a ClickHouse destination, ensure that you have:
- A ClickHouse instance reachable over HTTPS (ClickHouse Cloud or self-hosted)
- A table to insert into (see [Data format and table setup](#data-format-and-table-setup))
- A username and password with `INSERT` privileges on that table

## Data format and table setup

Ampersand delivers each read or subscribe message as a JSON object. The full payload schema is defined in [webhook.yaml](https://github.com/amp-labs/openapi/blob/main/webhook/webhook.yaml).

ClickHouse matches the payload's top-level keys to your table's columns by name. Create a column for each top-level key you want to store. Keys with no matching column are ignored. `result` is the array of synced records; store it as a JSON string.

```sql
CREATE TABLE ampersand_events (
action String,
objectName String,
operationId String,
installationId String,
operationTime String,
result String,
...
)
ENGINE = MergeTree()
ORDER BY tuple();
```

## Create a ClickHouse destination

Go to the [Destinations page](https://dashboard.withampersand.com/projects/_/destinations) in the Ampersand Dashboard and create a new ClickHouse destination.

You'll need to provide:

| Field | Type | Required | Example | Description |
|-------|------|----------|---------|-------------|
| **Destination name** | string | Yes | `ampersandClickHouse` | Alias to reference in your `amp.yaml` file |
| **URL** | string | Yes | `https://abc.clickhouse.cloud:8443` | HTTPS URL of your ClickHouse instance |
| **Username** | string | Yes | `default` | ClickHouse username |
| **Password** | string | Yes | | ClickHouse password |
| **Table name** | string | Yes | `ampersand_events` | Table to insert rows into |
| **Database** | string | No | `default` | Database containing the table |
| **Batch size** | integer | No | `1000` | Rows to buffer before flushing a batch (defaults to 1000) |
| **Max wait (seconds)** | integer | No | `30` | Max seconds to wait before flushing a batch (defaults to 30) |

## Refer to the destination in your integration

After creating your ClickHouse destination, reference it in your `amp.yaml` file:

```yaml
specVersion: 1.0.0
integrations:
- name: salesforceToClickHouse
displayName: Salesforce to ClickHouse
provider: salesforce
read:
objects:
- objectName: account
destination: ampersandClickHouse
- objectName: contact
destination: ampersandClickHouse
```
50 changes: 50 additions & 0 deletions src/destinations/eventbridge.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
---
title: Amazon EventBridge destinations
---

For more information on destinations, see the [Destinations](/destinations) page.

When new data is read from a SaaS instance via a [Read Action](/read-actions) or [Subscribe Action](/subscribe-actions), Ampersand publishes the payload as events to your Amazon EventBridge event bus.

## Prerequisites

Before setting up an EventBridge destination, ensure that you have:
- An AWS account with an EventBridge event bus
- AWS credentials with `events:PutEvents` permission on that event bus

## Data format

Ampersand publishes each read or subscribe message to your event bus. The message payload is carried in the event's `detail` field, and its schema is defined in [webhook.yaml](https://github.com/amp-labs/openapi/blob/main/webhook/webhook.yaml).

## Create an EventBridge destination

Go to the [Destinations page](https://dashboard.withampersand.com/projects/_/destinations) in the Ampersand Dashboard and create a new EventBridge destination.

You'll need to provide:

| Field | Type | Required | Example | Description |
|-------|------|----------|---------|-------------|
| **Destination name** | string | Yes | `ampersandEventBridge` | Alias to reference in your `amp.yaml` file |
| **Event bus name** | string | Yes | `default` | Name of your EventBridge event bus |
| **Region** | string | Yes | `us-west-2` | AWS region where your event bus is located |
| **AWS Access Key ID** | string | Yes | `AKIAIOSFODNN7EXAMPLE` | AWS access key with `events:PutEvents` permission |
| **AWS Secret Access Key** | string | Yes | | AWS secret access key |
| **Detail type** | string | No | `ampersand.record` | Value to set as the EventBridge `detail-type` on published events |

## Refer to the destination in your integration

After creating your EventBridge destination, reference it in your `amp.yaml` file:

```yaml
specVersion: 1.0.0
integrations:
- name: salesforceToEventBridge
displayName: Salesforce to EventBridge
provider: salesforce
read:
objects:
- objectName: account
destination: ampersandEventBridge
- objectName: contact
destination: ampersandEventBridge
```
4 changes: 0 additions & 4 deletions src/destinations/kinesis.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -83,10 +83,6 @@ You'll need to provide:
| **Partition key template** | string | No | [JMESPath](https://jmespath.org) template for partition key |
| **Endpoint URL** | string | No | Custom Kinesis endpoint if applicable (e.g. when using [LocalStack](https://www.localstack.cloud)) |

<Info>
Ampersand encrypts and stores your AWS credentials securely.
</Info>

## Refer to the destination in your integration

After creating your Kinesis destination, reference it in your `amp.yaml` file:
Expand Down
28 changes: 21 additions & 7 deletions src/destinations/overview.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: Overview
---

Ampersand currently supports webhook, Slack, Kinesis, and S3 destinations. Destinations allow you to route data synced from SaaS instances via [Read Actions](/read-actions) or [Subscribe Actions](/subscribe-actions), and to receive real-time [Notifications](/notifications) about important lifecycle events in your projects.
Ampersand supports webhook, Slack, Kinesis, and S3 destinations, along with a range of data warehouse and message queue destinations (ClickHouse, Snowflake, BigQuery, Redshift, Amazon SQS, Amazon SNS, Amazon EventBridge, Google Cloud Pub/Sub, and RabbitMQ). Destinations allow you to route data synced from SaaS instances via [Read Actions](/read-actions) or [Subscribe Actions](/subscribe-actions), and to receive real-time [Notifications](/notifications) about important lifecycle events in your projects.

## Add a destination to the Ampersand Dashboard

Expand Down Expand Up @@ -36,17 +36,31 @@ Destinations can also be used to receive [notifications](/notifications) about i

* [Webhook destinations](/destinations/webhooks)
* [Slack destinations](/destinations/slack)
* [Amazon Kinesis destinations](/destinations/kinesis)

### Blob storage

* [Amazon S3 destinations](/destinations/s3)

### Databases and warehouses

* [ClickHouse destinations](/destinations/clickhouse)
* [Snowflake destinations](/destinations/snowflake)
* [BigQuery destinations](/destinations/bigquery)
* [Amazon Redshift destinations](/destinations/redshift)

### Message queues and event buses

* [Amazon Kinesis destinations](/destinations/kinesis)
* [Amazon SQS destinations](/destinations/sqs)
* [Amazon SNS destinations](/destinations/sns)
* [Amazon EventBridge destinations](/destinations/eventbridge)
* [Google Cloud Pub/Sub destinations](/destinations/pubsub)
* [RabbitMQ destinations](/destinations/rabbitmq)

## Other Destinations

We have many other destination types on the roadmap, including:
We have more destination types on the roadmap, including:

* Postgres
* Ampersand-hosted Postgres
* Amazon SQS
* Google Cloud Storage
* Google PubSub
* RabbitMQ
* Azure Service Bus
48 changes: 48 additions & 0 deletions src/destinations/pubsub.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
title: Google Cloud Pub/Sub destinations
---

For more information on destinations, see the [Destinations](/destinations) page.

When new data is read from a SaaS instance via a [Read Action](/read-actions) or [Subscribe Action](/subscribe-actions), Ampersand publishes the payload as messages to your Google Cloud Pub/Sub topic.

## Prerequisites

Before setting up a Pub/Sub destination, ensure that you have:
- A Google Cloud project with a Pub/Sub topic
- A service account with the `Pub/Sub Publisher` role (or `pubsub.topics.publish` permission) on that topic, and a JSON key for it

## Data format

Ampersand publishes each read or subscribe message to your topic as JSON. The full payload schema is defined in [webhook.yaml](https://github.com/amp-labs/openapi/blob/main/webhook/webhook.yaml).

## Create a Pub/Sub destination

Go to the [Destinations page](https://dashboard.withampersand.com/projects/_/destinations) in the Ampersand Dashboard and create a new Pub/Sub destination.

You'll need to provide:

| Field | Type | Required | Example | Description |
|-------|------|----------|---------|-------------|
| **Destination name** | string | Yes | `ampersandPubSub` | Alias to reference in your `amp.yaml` file |
| **Project ID** | string | Yes | `my-gcp-project` | Google Cloud project ID |
| **Topic ID** | string | Yes | `ampersand-events` | Pub/Sub topic ID |
| **Service account credentials** | string | Yes | | Service account key JSON with publish access to the topic |

## Refer to the destination in your integration

After creating your Pub/Sub destination, reference it in your `amp.yaml` file:

```yaml
specVersion: 1.0.0
integrations:
- name: salesforceToPubSub
displayName: Salesforce to Pub/Sub
provider: salesforce
read:
objects:
- objectName: account
destination: ampersandPubSub
- objectName: contact
destination: ampersandPubSub
```
48 changes: 48 additions & 0 deletions src/destinations/rabbitmq.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
title: RabbitMQ destinations
---

For more information on destinations, see the [Destinations](/destinations) page.

When new data is read from a SaaS instance via a [Read Action](/read-actions) or [Subscribe Action](/subscribe-actions), Ampersand publishes the payload as messages to your RabbitMQ instance.

## Prerequisites

Before setting up a RabbitMQ destination, ensure that you have:
- A RabbitMQ instance reachable from the internet
- A connection URI with permission to publish
- A routing key that reaches a queue you can consume from

## Data format

Ampersand publishes each read or subscribe message to your RabbitMQ instance as JSON, using the routing key you configure. The full payload schema is defined in [webhook.yaml](https://github.com/amp-labs/openapi/blob/main/webhook/webhook.yaml).

## Create a RabbitMQ destination

Go to the [Destinations page](https://dashboard.withampersand.com/projects/_/destinations) in the Ampersand Dashboard and create a new RabbitMQ destination.

You'll need to provide:

| Field | Type | Required | Example | Description |
|-------|------|----------|---------|-------------|
| **Destination name** | string | Yes | `ampersandRabbitMQ` | Alias to reference in your `amp.yaml` file |
| **Connection URI** | string | Yes | `amqp://user:pass@host:5672` | AMQP connection URI (it embeds your credentials) |
| **Routing key** | string | Yes | `ampersand-events` | Routing key to publish messages with |

## Refer to the destination in your integration

After creating your RabbitMQ destination, reference it in your `amp.yaml` file:

```yaml
specVersion: 1.0.0
integrations:
- name: salesforceToRabbitMQ
displayName: Salesforce to RabbitMQ
provider: salesforce
read:
objects:
- objectName: account
destination: ampersandRabbitMQ
- objectName: contact
destination: ampersandRabbitMQ
```
Loading
Loading