tutorial

Very simple Spark interact to Kafka

Setup

1. Create python environment

Create python virtual environment and prepare data

python3 -m venv ./venv
source ./venv/bin/activate

Install package

pip3 install -r requirements.txt

Except all sh file

chmod +x script/*.sh

Download NASA logging data

./script/prepare.sh

2. Run kafka cluster written by docker

docker-compose up -d

3. Run simple demo spark

./script/run.sh