An end-to-end Azure Lakehouse data platform that ingests data from multiple sources (CSV, Nested JSON, and API payloads), processes it through a Medallion Architecture (Bronze, Silver, Gold) using Databricks and Delta Lake, and delivers analytics-ready data to Synapse Serverless SQL and Power BI. The pipeline includes schema standardization, data validation, deduplication, incremental MERGE with watermark tracking, quarantine handling, and pipeline logging to simulate a production-grade data engineering workflow.
- Multi-source ingestion from CSV, Nested JSON, and REST-style API payloads
- Azure Lakehouse architecture with Medallion design
- Incremental processing using Delta MERGE and watermark tracking
- Data quality validation, deduplication, and quarantine handling
- Star schema warehouse exposed through Synapse Serverless SQL
- Interactive Power BI dashboard for business reporting
This project simulates an IoT smart building environment by integrating data from multiple sources:
- CSV files
- Nested JSON files
- REST-style API payloads
The datasets include sensor readings, devices, users, and alerts, which are processed through a Medallion Architecture before being modeled into an analytics-ready warehouse.
An end-to-end Azure Lakehouse architecture that ingests multi-source data into ADLS Gen2, processes it through Databricks, stores curated datasets in Delta Lake, and serves analytics through Synapse Serverless SQL and Power BI.
The Medallion Architecture separates raw ingestion, data quality processing, and analytics-ready modeling to improve maintainability, reliability, and reporting performance.
The Gold layer implements a star schema consisting of a fact table and a device dimension to support analytical queries and Power BI reporting.
The Power BI dashboard provides interactive visualizations for sensor activity, power consumption, environmental conditions, and operational metrics built on the Gold warehouse layer.
| Component | Technology |
|---|---|
| Cloud | Microsoft Azure |
| Storage | Azure Data Lake Storage Gen2 |
| Processing | Azure Databricks |
| Engine | Apache Spark |
| Storage Format | Delta Lake |
| SQL Layer | Synapse Serverless SQL |
| Visualization | Power BI |
architecture/ Architecture diagrams
bronze/ Sample raw source data
notebooks/ Databricks pipelines
sql/ Synapse SQL views and QA queries
sample_data/ Small demo datasets
src/ Reusable pipeline modules
tests/ Validation scripts
- Multi-source ingestion (CSV, Nested JSON, API)
- Medallion Architecture (Bronze, Silver, Gold)
- Delta Lake MERGE for incremental loading
- Watermark-based processing
- Schema standardization and validation
- Duplicate detection and removal
- Quarantine for invalid records
- Pipeline execution logging and metrics
- Star schema data warehouse
- Power BI analytics dashboard
- Orchestrate pipelines using Apache Airflow
- Add CI/CD with GitHub Actions
- Expand the warehouse with additional fact and dimension tables
- Add automated data quality tests
- Support real-time ingestion with Apache Kafka