An Iceberg-lakehouse data-engineering lab built on the Atlas platform.
Curated Spark scenarios in Scala (Zeppelin) and PySpark (Jupyter), orchestrated with Airflow, plus Maven Scala Spark apps built by Jenkins — all over Apache Iceberg on MinIO, cataloged by the Atlas Iceberg REST catalog.
The lab implements a medallion lakehouse with four layers:
s3a://landing/ → bronze → silver → gold
(raw Parquet) (clean) (enriched) (aggregated/modelled)
Every table is an Apache Iceberg table, accessed through the Atlas Iceberg REST catalog (lakehouse). Compute is provided by a Spark cluster; Trino handles ad-hoc SQL and federated queries. Redpanda (Kafka-compatible) drives all streaming scenarios. Airflow schedules production DAGs; Jenkins builds and publishes Maven Spark apps to MinIO artifacts.
-
:material-database-search:{ .lg .middle } Scenario catalog
19 end-to-end Spark and Trino scenarios across bronze, silver, and gold layers.
-
:material-rocket-launch:{ .lg .middle } Spark apps
2 CI-verified Maven Scala Spark apps built by Jenkins and run by Airflow.
-
:material-table-large:{ .lg .middle } Datasets
5 curated datasets (NYC Taxi, TPC-H, Online Retail, GH Archive, Events) with
make datasets. -
:material-layers-triple:{ .lg .middle } Lakehouse design
Medallion layout, Iceberg namespaces, MinIO buckets, and the bronze smoke test.
-
:material-check-decagram:{ .lg .middle } Atlas platform
A1–A9 Atlas enablement checklist, expectations, and go-live runbook.
:octicons-arrow-right-24: Atlas enablement
-
:material-play-box-multiple:{ .lg .middle } Getting started
Prerequisites,
make datasets, starting the stack, and running notebooks.
| What | Count |
|---|---|
| Scenario notebooks (Scala + PySpark pairs) | 19 (14 batch, 4 streaming, 1 hybrid) |
| CI-verified Maven Spark apps | 2 |
| Curated datasets | 5 |
| Atlas enablement items (A1–A9) | 9 |
| Iceberg medallion layers | 3 (bronze / silver / gold) |
Batch Ingestion — batch_ingest
Medallion Pipeline — medallion
Data Quality — data_quality
Schema & Maintenance — schema_evolution, time_travel, table_maintenance
Streaming — streaming_ingest (events + gh_archive), streaming_windows, cdc_streaming
BI & Queries — federated_query, bi_query
Join Optimization — join_optimization
Dimensional Modeling — star_schema
Feature Engineering — feature_engineering
SCD — scd2
JSON Processing — json_flatten
Session Analysis — sessionization
!!! tip "New here?" Start with Getting started to get the stack running, then pick a scenario from the catalog or dive into the lakehouse design.
!!! info "Atlas platform" The Atlas platform underpins this lab. See Atlas enablement for the full A1–A9 checklist and Go-live runbook for production readiness steps.