dw-kafkaconnect

Kafka Connect image for data warehouse Snowflake sink workloads.

Available image

Images are published to all four JFrog registries on every merge to main. The tag format is <gitversion>-<strimzi_version>-<kafka_version>.

Strimzi version	Kafka version	Full image (replace <gitversion>)
0.43.0	3.7.0	tcgplayer-docker-{development,qa,staging,production}.jfrog.io/dw-kafkaconnect:<gitversion>-0.43.0-3.7.0

Bundled connector plugin:

Plugin	Version
Snowflake Kafka Connector (sink)	3.5.4

Worker-level shared libraries (not connector plugins):

Library	Version
AWS MSK IAM Auth	2.3.7
Confluent Avro / JSON Schema converters	7.7.7

Building images locally

Use build-local.sh to build the image for local testing. Tags follow the standard release format with local in place of the git version.

./build-local.sh

This produces:

dw-kafkaconnect:local-0.43.0-3.7.0

Reference this tag directly in another repository's Helm values or docker-compose for local integration testing.

Adding new connectors

Connectors must be put in their own directory under plugin.path for dependency isolation. This is achieved using Maven profiles to download related dependencies into their own folders. Only pom.xml and Dockerfile need to be updated using the same conventions.

Example Helm Configuration

See example-helm-values.yaml for a minimal Helm deployment configuration.

Connector smoke tests

This repository includes lightweight smoke tests under smoke-tests/.

Smoke tests prove that:

• Kafka Connect starts in a disposable local environment.
• /connector-plugins exposes the Snowflake sink connector class.
• Kafka Connect connector config validation executes for Snowflake.

Smoke tests do not prove:

• application-level compatibility with downstream services
• production performance or reliability characteristics
• real Snowflake connectivity in required PR checks

Run smoke tests locally

bash smoke-tests/run-all-versions.sh

Build an image variant:

docker build \
  --build-arg STRIMZI_VERSION=0.43.0 \
  --build-arg KAFKA_VERSION=3.7.0 \
  --build-arg POM_DIR=versions/strimzi-0.43.0-kafka-3.7.0 \
  -t dw-kafkaconnect:local-0.43.0-3.7.0 .

Run plugin discovery:

CONNECT_IMAGE=dw-kafkaconnect:local-0.43.0-3.7.0 \
  smoke-tests/scripts/assert-plugins.sh

Run connector config validation:

CONNECT_IMAGE=dw-kafkaconnect:local-0.43.0-3.7.0 \
  smoke-tests/scripts/smoke-snowflake-config.sh

Optional Snowflake end-to-end testing

Real Snowflake end-to-end smoke testing should run in a separate manual or scheduled workflow with secrets, not in required PR checks.

Running the image locally with Docker

The Strimzi images are configured by the operator when running in Kubernetes. Up until 0.45.x you could run the image locally without the operator without any issues, however this was changed with the configuration being done in the operator.

kafka_connect_config_generator.sh is the script that was used to generate the configs and is copied into the container to maintain compatibility.

To run the script change the command of the container to:

  kafka-connect:
    image: tcgplayer-docker-development-local.jfrog.io/dw-kafkaconnect
    hostname: connect
    ports:
      - "8083:8083"
    healthcheck:
      test: ["CMD", "curl", "--output", "/dev/null", "--silent", "--head", "--fail", "http://localhost:8083/connectors"]
      interval: 30s
      timeout: 10s
      retries: 10
    command: sh -c "./kafka_connect_config_generator.sh | tee /opt/kafka/custom-config/kafka-connect.properties && ./kafka_connect_run.sh"

Updating

There are three things to consider when updating:

• The version of Strimzi
• The version of Kafka we are connecting to
• The version of each plugin

The Strimzi docker image has the format ${strimzi version}-kafka-${kafka version}. See the Strimzi downloads page for compatibility. Plugin version compatibility with Kafka will need testing as Kafka does not produce a BOM that we can use to pin shared dependencies.

Local testing will not be sufficient; we will also need to test in CP2 due to Strimzi having a different method of configuring Kafka Connect.