I'm a Data Engineer specialized in designing and building production-grade, cloud-native data pipelines that transform raw data into actionable business intelligence. My expertise spans multiple cloud platforms and data orchestration frameworks, with a passion for building scalable, reliable systems that solve real-world business problems.

I excel at:

• 🏗️ End-to-end ETL/ELT pipeline design using modern tools (Airflow, Databricks, Snowflake)
• ☁️ Cloud-native architecture on Azure & AWS with strong data warehouse expertise
• 📊 Data modeling & analytics using medallion architecture and dimensional modeling
• 🔐 Data governance with Delta Lake, Unity Catalog, and security best practices
• ⚙️ Workflow orchestration using Apache Airflow with Docker containerization
• 📈 BI & analytics integrating Snowflake/Databricks with Power BI for business insights

A production-grade data engineering solution that ingests, transforms, and analyzes End-of-Day securities pricing data at scale.

Business Impact:

• ✅ 87.5% faster ingestion - Reduced from 4 hours (manual) to 30 minutes (automated)
• ✅ 2+ hours faster insights - Eliminated CSV bottlenecks for trading teams
• ✅ 99.5% pipeline uptime - Automated monitoring & error handling
• ✅ 80% error reduction - Data quality validation at every layer
• ✅ 5,000+ daily records - Processing 26,000+ trades with 542 unique securities

Tech Stack:

• Orchestration: Apache Airflow 2.8.0 (Docker)
• Data Warehouse: Snowflake (4-layer: RAW → CORE → DIM → FACT → SA)
• Cloud Storage: AWS S3 (Bronze layer)
• APIs: Massive Stock Market API integration
• BI & Reporting: Power BI (2 dashboards, 6 analytics views)
• Alerting: Slack notifications on success/failure

Architecture:

Massive API  →  Airflow DAG  →  AWS S3  →  Snowflake ETL  →  Power BI Dashboards
  (5K/day)     (Download)     (Stage)    (Transform)         (Analytics)

Key Components:

• Daily Data Pipeline: Automated stock price ingestion via Massive API
• 4-Layer Data Warehouse: RAW (immutable) → CORE (cleansed) → DIM (dimensions) → FACT (analytics-ready)
• 6 Subject Area Views: Market liquidity, equity performance, sector analysis, watchlist insights
• Interactive Dashboards: Real-time trading insights, liquidity monitoring, volatility trends
• Error Handling: Reject tables, validation checks, automated alerts

📌 GitHub Repository

Cloud-native ETL pipeline leveraging Azure Databricks, Delta Lake, and Power BI for real-time ecommerce analytics.

Highlights:

• Medallion Architecture: Bronze → Silver → Gold data transformation layers
• Unified Governance: Azure Managed Identity + Unity Catalog for secure, credentialless ADLS access
• Data Reliability: Delta Lake time-travel & ACID transactions across all layers
• Multi-Domain Processing: Order Items, Order Returns, Order Shipments
• BI Integration: Power BI dashboards for real-time business metrics

Tech Stack:

• Azure Databricks | PySpark | Delta Lake | Unity Catalog | Azure Data Lake | Power BI

📌 GitHub Repository

• 🏆 Built production-grade pipelines processing 5,000+ daily records with 99.5% uptime
• 🏆 Reduced data latency by 87.5% through intelligent orchestration and optimization
• 🏆 Designed scalable data warehouses serving 2+ executive dashboards and 6+ analytics views
• 🏆 Implemented data governance with Unity Catalog and Delta Lake across multi-cloud platforms
• 🏆 Created automated monitoring with Slack alerts and data quality validation
• 🏆 Optimized Spark jobs improving query performance by 45% through best practices

• 🔍 Advanced Spark optimization techniques (partitioning, bucketing, caching strategies)
• 🔍 Data lakehouse architecture patterns and best practices
• 🔍 Real-time streaming with Kafka & Spark Structured Streaming
• 🔍 Cost optimization for cloud data platforms
• 🔍 Data observability & monitoring tools and frameworks

I'm always interested in discussing data engineering challenges, cloud architecture, and building systems that scale.

Building scalable data systems | Cloud-native architectures | Real-world impact through data