English · Русский

connector-cdc

a learning project with 3 connector builds for kafka connect that:

• read changes from postgres;
• send events to kafka;
• can save the result to a file.

structure

• custom — the connector's java code;
• connector-plugins — configs for kafka connect;
• docker-compose.yaml — a local setup with postgres, kafka, schema registry, and connect.

3 implementations

1. fully custom with polling

   • every few seconds the app reads new rows from the table by id
   • after sending to kafka it saves the offset to a file

2. fully custom streaming

   • the app reads the postgres wal with logical replication
   • events go to kafka almost right away, with no polling
   • a mode for streaming cdc without the kafka connect api

3. streaming with the kafka connect api

   • uses sourceconnector and sourcetask from the kafka connect api
   • the wal is read inside here too, but kafka connect handles startup and lifecycle
   • needed when you want a full kafka connect plugin

dependencies

• docker and docker compose
• java 17
• maven

start

1. docker compose up -d
2. cd custom && mvn clean package
3. pick a run mode:
   • custom builds: run the app's main class from the custom module
   • with the kafka connect api: put the jar into connector-plugins and start kafka-connect