Skip to content

feat(destinations): add batchSize/maxWaitSecs to warehouse destination metadata (ENG-4130) #395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
RajatPawar merged 1 commit into from
Aug 18, 2026
Merged

feat(destinations): add batchSize/maxWaitSecs to warehouse destination metadata (ENG-4130) #395

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 40 additions & 0 deletions api/api.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3230,6 +3230,20 @@ paths:
description: The routing key for the `rabbitmq` destination.
example: ampersand.events
x-go-type-skip-optional-pointer: true
batchSize:
type: integer
description: For warehouse destinations (`clickhouse`, `snowflake`,
`bigquery`, `redshift`), the number of rows to buffer before flushing
a batch. Defaults to 1000.
example: 1000
x-go-type-skip-optional-pointer: true
maxWaitSecs:
type: integer
description: For warehouse destinations, the maximum number of seconds to
wait before flushing a batch (whichever comes first with `batchSize`).
Defaults to 30.
example: 30
x-go-type-skip-optional-pointer: true
responses:
201:
description: Created
Expand Down Expand Up @@ -3401,6 +3415,8 @@ paths:
- metadata.detailType
- metadata.topicId
- metadata.routingKey
- metadata.batchSize
- metadata.maxWaitSecs
- secrets.awsKeyId
- secrets.awsSecretKey
- secrets.awsSessionToken
Expand Down Expand Up @@ -3541,6 +3557,14 @@ paths:
type: string
description: The routing key for the `rabbitmq` destination.
example: ampersand.events
batchSize:
type: integer
description: For warehouse destinations, rows to buffer before flushing a batch (default 1000).
example: 1000
maxWaitSecs:
type: integer
description: For warehouse destinations, max seconds before flushing a batch (default 30).
example: 30
secrets:
type: object
description: Secrets for the destination.
Expand Down Expand Up @@ -5410,6 +5434,14 @@ components:
type: string
description: The routing key for the rabbitmq destination
example: ampersand.events
batchSize:
type: integer
description: For warehouse destinations, rows to buffer before flushing a batch (default 1000)
example: 1000
maxWaitSecs:
type: integer
description: For warehouse destinations, max seconds before flushing a batch (default 30)
example: 30
createTime:
type: string
description: The time the destination was created.
Expand Down Expand Up @@ -5558,6 +5590,14 @@ components:
type: string
description: The routing key for the rabbitmq destination
example: ampersand.events
batchSize:
type: integer
description: For warehouse destinations, rows to buffer before flushing a batch (default 1000)
example: 1000
maxWaitSecs:
type: integer
description: For warehouse destinations, max seconds before flushing a batch (default 30)
example: 30
createTime:
type: string
description: The time the destination was created.
Expand Down
44 changes: 43 additions & 1 deletion api/generated/api.bundled.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4733,6 +4733,18 @@
"description": "The routing key for the `rabbitmq` destination.",
"example": "ampersand.events",
"x-go-type-skip-optional-pointer": true
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations (`clickhouse`, `snowflake`, `bigquery`, `redshift`), the number of rows to buffer before flushing a batch. Defaults to 1000.",
"example": 1000,
"x-go-type-skip-optional-pointer": true
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, the maximum number of seconds to wait before flushing a batch (whichever comes first with `batchSize`). Defaults to 30.",
"example": 30,
"x-go-type-skip-optional-pointer": true
}
}
}
Expand Down Expand Up @@ -4955,7 +4967,7 @@
"items": {
"type": "string"
},
"description": "Array of field paths specifying which fields to update. Allowed values include:\n- name\n- metadata.url\n- metadata.headers\n- metadata.region\n- metadata.streamName\n- metadata.endpointUrl\n- metadata.partitionKeyTemplate\n- metadata.bucket\n- metadata.keyTemplate\n- metadata.storageClass\n- metadata.username\n- metadata.tableName\n- metadata.database\n- metadata.accountIdentifier\n- metadata.userId\n- metadata.dbName\n- metadata.schemaName\n- metadata.projectId\n- metadata.datasetId\n- metadata.tableId\n- metadata.workgroupName\n- metadata.clusterIdentifier\n- metadata.dbUser\n- metadata.account\n- metadata.container\n- metadata.queueUrl\n- metadata.topicArn\n- metadata.eventBusName\n- metadata.detailType\n- metadata.topicId\n- metadata.routingKey\n- secrets.awsKeyId\n- secrets.awsSecretKey\n- secrets.awsSessionToken\n- secrets.password\n- secrets.privateKey\n- secrets.credentials\n- secrets.accessKeyId\n- secrets.secretAccessKey\n- secrets.accessKey\n- secrets.uri\n",
"description": "Array of field paths specifying which fields to update. Allowed values include:\n- name\n- metadata.url\n- metadata.headers\n- metadata.region\n- metadata.streamName\n- metadata.endpointUrl\n- metadata.partitionKeyTemplate\n- metadata.bucket\n- metadata.keyTemplate\n- metadata.storageClass\n- metadata.username\n- metadata.tableName\n- metadata.database\n- metadata.accountIdentifier\n- metadata.userId\n- metadata.dbName\n- metadata.schemaName\n- metadata.projectId\n- metadata.datasetId\n- metadata.tableId\n- metadata.workgroupName\n- metadata.clusterIdentifier\n- metadata.dbUser\n- metadata.account\n- metadata.container\n- metadata.queueUrl\n- metadata.topicArn\n- metadata.eventBusName\n- metadata.detailType\n- metadata.topicId\n- metadata.routingKey\n- metadata.batchSize\n- metadata.maxWaitSecs\n- secrets.awsKeyId\n- secrets.awsSecretKey\n- secrets.awsSessionToken\n- secrets.password\n- secrets.privateKey\n- secrets.credentials\n- secrets.accessKeyId\n- secrets.secretAccessKey\n- secrets.accessKey\n- secrets.uri\n",
"example": [
"name",
"metadata.url",
Expand Down Expand Up @@ -5121,6 +5133,16 @@
"type": "string",
"description": "The routing key for the `rabbitmq` destination.",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000).",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30).",
"example": 30
}
}
},
Expand Down Expand Up @@ -7992,6 +8014,16 @@
"type": "string",
"description": "The routing key for the rabbitmq destination",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000)",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30)",
"example": 30
}
}
},
Expand Down Expand Up @@ -8178,6 +8210,16 @@
"type": "string",
"description": "The routing key for the rabbitmq destination",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000)",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30)",
"example": 30
}
}
},
Expand Down
84 changes: 83 additions & 1 deletion api/generated/api.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49245,6 +49245,18 @@
"description": "The routing key for the `rabbitmq` destination.",
"example": "ampersand.events",
"x-go-type-skip-optional-pointer": true
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations (`clickhouse`, `snowflake`, `bigquery`, `redshift`), the number of rows to buffer before flushing a batch. Defaults to 1000.",
"example": 1000,
"x-go-type-skip-optional-pointer": true
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, the maximum number of seconds to wait before flushing a batch (whichever comes first with `batchSize`). Defaults to 30.",
"example": 30,
"x-go-type-skip-optional-pointer": true
}
}
}
Expand Down Expand Up @@ -49438,6 +49450,16 @@
"type": "string",
"description": "The routing key for the rabbitmq destination",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000)",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30)",
"example": 30
}
}
},
Expand Down Expand Up @@ -50447,6 +50469,16 @@
"type": "string",
"description": "The routing key for the rabbitmq destination",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000)",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30)",
"example": 30
}
}
},
Expand Down Expand Up @@ -50839,6 +50871,16 @@
"type": "string",
"description": "The routing key for the rabbitmq destination",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000)",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30)",
"example": 30
}
}
},
Expand Down Expand Up @@ -51371,7 +51413,7 @@
"items": {
"type": "string"
},
"description": "Array of field paths specifying which fields to update. Allowed values include:\n- name\n- metadata.url\n- metadata.headers\n- metadata.region\n- metadata.streamName\n- metadata.endpointUrl\n- metadata.partitionKeyTemplate\n- metadata.bucket\n- metadata.keyTemplate\n- metadata.storageClass\n- metadata.username\n- metadata.tableName\n- metadata.database\n- metadata.accountIdentifier\n- metadata.userId\n- metadata.dbName\n- metadata.schemaName\n- metadata.projectId\n- metadata.datasetId\n- metadata.tableId\n- metadata.workgroupName\n- metadata.clusterIdentifier\n- metadata.dbUser\n- metadata.account\n- metadata.container\n- metadata.queueUrl\n- metadata.topicArn\n- metadata.eventBusName\n- metadata.detailType\n- metadata.topicId\n- metadata.routingKey\n- secrets.awsKeyId\n- secrets.awsSecretKey\n- secrets.awsSessionToken\n- secrets.password\n- secrets.privateKey\n- secrets.credentials\n- secrets.accessKeyId\n- secrets.secretAccessKey\n- secrets.accessKey\n- secrets.uri\n",
"description": "Array of field paths specifying which fields to update. Allowed values include:\n- name\n- metadata.url\n- metadata.headers\n- metadata.region\n- metadata.streamName\n- metadata.endpointUrl\n- metadata.partitionKeyTemplate\n- metadata.bucket\n- metadata.keyTemplate\n- metadata.storageClass\n- metadata.username\n- metadata.tableName\n- metadata.database\n- metadata.accountIdentifier\n- metadata.userId\n- metadata.dbName\n- metadata.schemaName\n- metadata.projectId\n- metadata.datasetId\n- metadata.tableId\n- metadata.workgroupName\n- metadata.clusterIdentifier\n- metadata.dbUser\n- metadata.account\n- metadata.container\n- metadata.queueUrl\n- metadata.topicArn\n- metadata.eventBusName\n- metadata.detailType\n- metadata.topicId\n- metadata.routingKey\n- metadata.batchSize\n- metadata.maxWaitSecs\n- secrets.awsKeyId\n- secrets.awsSecretKey\n- secrets.awsSessionToken\n- secrets.password\n- secrets.privateKey\n- secrets.credentials\n- secrets.accessKeyId\n- secrets.secretAccessKey\n- secrets.accessKey\n- secrets.uri\n",
"example": [
"name",
"metadata.url",
Expand Down Expand Up @@ -51547,6 +51589,16 @@
"type": "string",
"description": "The routing key for the `rabbitmq` destination.",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000).",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30).",
"example": 30
}
}
},
Expand Down Expand Up @@ -51798,6 +51850,16 @@
"type": "string",
"description": "The routing key for the rabbitmq destination",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000)",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30)",
"example": 30
}
}
},
Expand Down Expand Up @@ -69460,6 +69522,16 @@
"type": "string",
"description": "The routing key for the rabbitmq destination",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000)",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30)",
"example": 30
}
}
},
Expand Down Expand Up @@ -69656,6 +69728,16 @@
"type": "string",
"description": "The routing key for the rabbitmq destination",
"example": "ampersand.events"
},
"batchSize": {
"type": "integer",
"description": "For warehouse destinations, rows to buffer before flushing a batch (default 1000)",
"example": 1000
},
"maxWaitSecs": {
"type": "integer",
"description": "For warehouse destinations, max seconds before flushing a batch (default 30)",
"example": 30
}
}
},
Expand Down
Loading