feat: implement Kafka-to-Spark ingestion pipeline (Chapter 4) - #3

Merged

JayKay24 merged 6 commits into

scalable-pipeline-architecture

Jun 18, 2026

Commits on Jun 18, 2026

feat: implement Kafka-to-Spark ingestion pipeline with Docker and producer utilities
JayKay24
committed
03184d9
docs: add notebook cell markers and docstrings to Kafka ingestion script
JayKay24
committed
3379e49
docs: add project-level documentation for essentials and ingestion modules in the main repository README
JayKay24
committed
2f47b0e
feat: add descriptive comments to Kafka-to-file ingestion job process
JayKay24
committed
c004802
feat: add robust error handling to producer and support for both batch and streaming modes in Spark job
JayKay24
committed
19e593f
refactor: add type hints to load_config and improve interruption handling and partitioning comments in Spark job
JayKay24
committed
a414e3e