tutorial

Very simple Spark interact to Kafka

Setup

1. Create python environment

Create python virtual environment and prepare data

python3 -m venv ./venv
source ./venv/bin/activate

Install package

pip3 install -r requirements.txt

Create .env file with same .env.sample format. You can copy .env.sample by following command

cp .env.sample .env

Except all sh file

chmod +x script/*.sh

Download NASA logging data

./script/prepare.sh

2. Run kafka cluster written by docker

docker-compose up -d

3. Run simple demo spark

./script/run.sh