FluxPipe is a fully containerized, production-ready data platform that demonstrates end-to-end data engineering across the complete modern stack. It combines batch ingestion, real-time stream processing, a star-schema data warehouse, ML experiment tracking, a .NET 8 REST API, and full observability — all coordinated through 20 Docker services driven by a single docker compose command.
Designed to reflect production engineering standards: multi-layered testing, blue-green and canary deployment strategies, structured logging, Polly-based resilience, and multi-cloud Kubernetes support across AWS EKS, GCP GKE, and Azure AKS.
- Platform at a Glance
- Architecture
- Pipeline Flows
- Technology Stack
- Prerequisites
- Quick Start
- Make Commands Reference
- Deployment
- Service URLs
- REST API — .NET 8 Backend
- Data Warehouse Schema
- Airflow DAGs
- Testing
- CI/CD Pipeline
- Project Structure
- Configuration
- Contributing
- License
| Dimension | Detail |
|---|---|
| Services | 20 containerized services, single docker compose stack |
| Pipelines | 3 Airflow DAGs — daily batch, 15-min streaming monitor, hourly warehouse ETL |
| API Surface | 20 REST endpoints across 8 controllers (.NET 8 / ASP.NET Core) |
| Test Coverage | 35 unit and integration tests across 6 test files |
| Warehouse | Star schema — 4 dimensions, 3 facts, 2 aggregation tables |
| Deployment Targets | Local Docker, AWS EKS, GCP GKE, Azure AKS, on-prem Kubernetes |
| IaC | Terraform (AWS VPC + EKS + RDS + S3) + Helm chart with per-provider values |
| Deployment Strategies | Rolling, blue-green, canary (Argo Rollouts) |
| Languages | Python, C#, Shell, HCL, SQL, JavaScript, HTML/CSS |
The platform is structured into six functional zones: data sources, orchestration, stream ingestion, processing, storage, and observability. All services communicate over a shared Docker network and can be redeployed to any Kubernetes cluster using the included Helm chart.
┌─────────────────────────────────────────────────────────────────┐
│ Sources │ Orchestration │ Streaming │
│ MySQL 8.0 │ Airflow 2.7.3 │ Zookeeper → Kafka 7.5 │
│ Kafka Producer │ 3 DAGs │ sensor_readings topic │
├─────────────────────────────────────────────────────────────────┤
│ Processing │ Storage │
│ Great Expectations (validation) │ PostgreSQL 15 (warehouse)│
│ Spark Master + Worker │ MinIO (S3-compatible) │
│ │ MongoDB 6 / Redis 7 │
│ │ InfluxDB 2.7 │
│ │ Elasticsearch 8.11 │
├─────────────────────────────────────────────────────────────────┤
│ Serving │ Observability │
│ .NET 8 API + Swagger │ Prometheus → Grafana │
│ MLflow 2.9.2 (experiment registry) │ Serilog (structured log) │
│ Snowflake (primary warehouse) │ Request-ID correlation │
└─────────────────────────────────────────────────────────────────┘
Full architecture documentation is available in ARCHITECTURE.md.
MySQL → Airflow DAG → Great Expectations → MinIO (raw)
→ Spark ETL → PostgreSQL (processed)
Kafka Producer → sensor_readings topic → Spark Streaming
→ Anomaly Detection → PostgreSQL + MinIO
Staging Tables → Dimension Load (customers, products, dates, devices)
→ Fact Load (orders, sensors, pipeline runs)
→ Aggregations (daily orders, hourly sensors)
| Layer | Technology | Version | Purpose |
|---|---|---|---|
| Orchestration | Apache Airflow | 2.7.3 | DAG scheduling and pipeline management |
| Batch Processing | Apache Spark | 3.5.3 | Large-scale ETL and transformation |
| Stream Ingestion | Apache Kafka | 7.5.0 (Confluent) | Real-time event streaming |
| Data Quality | Great Expectations | latest | Schema validation and quality checks |
| Source Database | MySQL | 8.0 | Transactional source system |
| Data Warehouse | Snowflake / PostgreSQL | — / 15 | Star-schema warehouse (Snowflake primary, PG fallback) |
| Object Storage | MinIO | latest | S3-compatible data lake |
| Cache | Redis | 7-alpine | Session storage and caching |
| Document Store | MongoDB | 6.0.13 | Semi-structured NoSQL storage |
| Time Series | InfluxDB | 2.7 | IoT sensor and metrics storage |
| Search | Elasticsearch | 8.11.3 | Full-text search and log indexing |
| REST API | .NET 8 (ASP.NET Core) | 8.0 | Backend API with Swagger documentation |
| ML Tracking | MLflow | 2.9.2 | Experiment tracking and model registry |
| Metrics | Prometheus | 2.48.1 | Metrics collection and alerting rules |
| Dashboards | Grafana | 10.2.3 | Monitoring and visualization |
| Governance | Apache Atlas (stub) | — | Data lineage registration |
| IaC | Terraform + Kubernetes | — | Cloud deployment and infrastructure |
| CI/CD | GitHub Actions | — | Lint, test, build, integration gate |
- Docker and Docker Compose v2+
- Python 3.10+ (for local test execution)
- GNU Make
- 16 GB RAM recommended for the full stack; 8 GB is sufficient with
make up-lite - The following ports must be available:
3000, 3306, 5000, 5001, 5432, 6379, 7077, 8080, 8081, 8086, 9000, 9001, 9090, 9092, 9200, 27017
# 1. Clone the repository
git clone https://github.com/aryangahlot50/FluxPipe.git
cd FluxPipe
# 2. Create the environment file
cp .env.example .env
# 3. Build and start all 20 services
make build
make up
# 4. Confirm services are healthy
make health
make urls
# 5. Trigger the batch ingestion pipeline
make trigger-batch
# 6. Trigger the warehouse ETL
make trigger-warehouse
# 7. Submit Spark jobs directly
make spark-batch
make spark-streamFor a lighter footprint (approximately 8 GB RAM), use make up-lite, which starts the 16 core services only.
| Command | Description |
|---|---|
make up |
Start all 20 services (~16 GB RAM) |
make up-lite |
Start core services only (~8 GB RAM) |
make down |
Stop all services |
make build |
Build all Docker images |
make rebuild |
Full rebuild from scratch (no cache) |
make test |
Run 35 Python tests via pytest |
make lint |
Lint Python code with flake8 |
make health |
Show container health status |
make status |
Show running container list |
make urls |
Print all service access URLs |
make spark-batch |
Submit the Spark batch ETL job |
make spark-stream |
Submit the Spark streaming job |
make trigger-batch |
Trigger batch_ingestion_dag in Airflow |
make trigger-warehouse |
Trigger warehouse_transform_dag in Airflow |
make list-dags |
List all registered Airflow DAGs |
make kafka-topics |
List Kafka topics |
make logs-kafka |
Tail Kafka container logs |
make clean |
Stop services and remove all volumes |
make format |
Format code across Python, C#, HTML/CSS/JS |
make format-check |
Check formatting without writing changes |
make deploy-local |
Deploy via Docker Compose (full stack) |
make deploy-lite |
Deploy via Docker Compose (lite, 8 GB) |
make deploy-k8s |
Deploy to any Kubernetes cluster via Helm |
make deploy-aws |
Deploy to AWS EKS using Terraform + Helm |
make deploy-gcp |
Deploy to GCP GKE via Helm |
make deploy-azure |
Deploy to Azure AKS via Helm |
make deploy-onprem |
Deploy to on-prem Kubernetes (k3s, kubeadm) |
make deploy-teardown |
Remove deployment from any target |
FluxPipe can be deployed to any environment with a single command. The Helm chart at helm/fluxpipe/ provides per-provider values files for each supported target.
| Target | Command | Requirements | Resources |
|---|---|---|---|
| Local (full) | make deploy-local |
Docker | 16 GB RAM, 14 vCPU |
| Local (lite) | make deploy-lite |
Docker | 8 GB RAM, 7 vCPU |
| Any Kubernetes | make deploy-k8s |
kubectl, Helm | K8s cluster |
| AWS EKS | make deploy-aws |
Terraform, AWS CLI | EKS cluster |
| GCP GKE | make deploy-gcp |
gcloud, Helm | GKE cluster |
| Azure AKS | make deploy-azure |
az CLI, Helm | AKS cluster |
| On-premises | make deploy-onprem |
kubectl, Helm | k3s / kubeadm / Rancher |
helm repo add bitnami https://charts.bitnami.com/bitnami
helm repo update
helm install fluxpipe ./helm/fluxpipe \
-f helm/fluxpipe/values-aws.yaml \
--set postgresql.auth.password=YOUR_PASSWORD \
--set minio.auth.rootPassword=YOUR_PASSWORD \
--namespace pipeline --create-namespaceSwap values-aws.yaml for values-gcp.yaml, values-azure.yaml, or values-onprem.yaml as needed.
Provisions a full AWS environment — VPC with public and private subnets, NAT Gateway, EKS with autoscaling node groups, RDS PostgreSQL (encrypted, multi-AZ), and an S3 data lake with versioning and lifecycle policies.
cd terraform
cp terraform.tfvars.example terraform.tfvars
# Edit terraform.tfvars with your account settings
terraform init && terraform plan && terraform apply| Service | URL | Default Credentials |
|---|---|---|
| Airflow UI | http://localhost:8080 | admin / airflow_admin_2024 |
| Grafana | http://localhost:3000 | admin / admin_secret_2024 |
| MinIO Console | http://localhost:9001 | minio / minio_secret_2024 |
| MLflow UI | http://localhost:5001 | — |
| Spark Master UI | http://localhost:8081 | — |
| Swagger (.NET API) | http://localhost:5000/swagger | — |
| Prometheus | http://localhost:9090 | — |
| Elasticsearch | http://localhost:9200 | — |
| Kafka | localhost:9092 |
— |
| PostgreSQL | localhost:5432 |
pipeline_user / pipeline_secret_2024 |
| MySQL | localhost:3306 |
pipeline_user / pipeline_secret_2024 |
| MongoDB | localhost:27017 |
— |
| Redis | localhost:6379 |
— |
| InfluxDB | http://localhost:8086 | — |
The API runs on port 5000 with interactive Swagger documentation at /swagger. It is built on ASP.NET Core with Serilog structured logging, Polly retry policies (3x exponential backoff), and Dapper as a lightweight micro-ORM.
| Layer | Components | Technology |
|---|---|---|
| Controllers | Batch, Streaming, Warehouse, Insights, ML, Governance, CI, Monitoring | ASP.NET Core |
| Services | 11 services with interfaces: Db, Kafka, MinIO, Batch, Streaming, Atlas, MLflow, GE, CI, Monitoring, Insights | Dapper, Confluent.Kafka, AWS SDK |
| Health Checks | MySQL, PostgreSQL, Kafka, MinIO, Airflow, MLflow | ASP.NET Health Checks |
| Configuration | 8 validated options classes with ValidateOnStart() |
Options Pattern |
| Resilience | Polly retry (3x exponential backoff), request timeouts | Polly |
| Logging | Console + file sinks, request-ID correlation across all services | Serilog |
| Method | Endpoint | Controller | Description |
|---|---|---|---|
POST |
/api/batch/ingest |
BatchController | Extract from MySQL, validate, upload to MinIO, trigger Airflow |
POST |
/api/stream/produce |
StreamingController | Produce message to a Kafka topic |
POST |
/api/stream/run |
StreamingController | Trigger the streaming monitoring DAG |
POST |
/api/warehouse/transform |
WarehouseController | Trigger Snowflake warehouse ETL |
GET |
/api/warehouse/health |
WarehouseController | Check warehouse and Snowflake connectivity |
GET |
/api/warehouse/snowflake/status |
WarehouseController | Snowflake configuration status and schema info |
GET |
/api/warehouse/aggregations/daily-orders |
WarehouseController | Live daily order aggregations |
GET |
/api/warehouse/pipeline-runs |
WarehouseController | Pipeline run history |
GET |
/api/insights/overview |
InsightsController | Composite dashboard: health and warehouse metrics |
GET |
/api/insights/pipeline-runs |
InsightsController | Recent runs from fact_pipeline_runs |
GET |
/api/insights/daily-orders |
InsightsController | Daily order trends from agg_daily_orders |
GET |
/api/insights/sensor-summary |
InsightsController | Hourly sensor statistics and anomaly counts |
POST |
/api/governance/lineage |
GovernanceController | Register data lineage in Apache Atlas |
POST |
/api/ml/run |
MLController | Create an MLflow experiment run |
POST |
/api/ci/trigger |
CIController | Dispatch a GitHub Actions workflow |
GET |
/api/monitor/health |
MonitoringController | Aggregated health status across all services |
GET |
/health |
Built-in | Full health check with 6 dependency probes |
GET |
/health/ready |
Built-in | Readiness probe (critical dependencies only) |
GET |
/health/live |
Built-in | Liveness probe (always 200) |
GET |
/swagger |
Swashbuckle | Interactive API documentation |
The warehouse uses a star schema backed by Snowflake as the primary store, with PostgreSQL available as a local development fallback. When SNOWFLAKE_ACCOUNT is configured, data is routed through staging tables into Snowflake automatically.
| Table | Description |
|---|---|
dim_customers |
Customer master data |
dim_products |
Product catalog |
dim_date |
Calendar date dimension |
dim_devices |
IoT device registry |
| Table | Description |
|---|---|
fact_orders |
Transactional orders linked to customer, product, and date |
fact_sensor_readings |
IoT measurements linked to device and date |
fact_pipeline_runs |
Pipeline execution history and status |
| Table | Description |
|---|---|
agg_daily_orders |
Daily order totals and revenue summaries |
agg_hourly_sensors |
Hourly sensor averages and reading counts |
| DAG | Schedule | Description |
|---|---|---|
batch_ingestion_dag |
Daily | Extract from MySQL → validate with Great Expectations → upload raw data to MinIO → Spark transform → load to PostgreSQL |
streaming_monitoring_dag |
Every 15 min | Monitor Kafka broker health, check consumer lag, alert on anomalies |
warehouse_transform_dag |
Hourly | Stage data in Snowflake, load dimensions and facts, refresh aggregations (PG fallback when Snowflake is not configured) |
The test suite contains 35 tests across 6 files, covering infrastructure configuration, message serialization, data quality rules, warehouse SQL integrity, and Snowflake integration.
make test| File | Scope |
|---|---|
test_pipeline_config.py |
Environment variables, connection strings, service configuration |
test_kafka_producer.py |
Kafka producer logic, message serialization, topic configuration |
test_data_validation.py |
Great Expectations suite, schema validation, data quality rules |
test_warehouse_sql.py |
Warehouse DDL, star-schema integrity, aggregation queries |
test_snowflake.py |
Snowflake SQL schema, connector module, DAG/BI/API integration |
test_docker_infrastructure.py |
Docker Compose structure, service definitions, port mappings |
The GitHub Actions workflow (.github/workflows/cicd-pipeline.yml) executes on every push and pull request targeting master or main.
| Stage | Job | Details |
|---|---|---|
| Lint | lint |
flake8, black (formatting), isort (import ordering) |
| Test | python-tests |
35 unit tests with pytest |
| Build | docker-build |
Build matrix: airflow, spark, kafka-producer, dotnet-api |
| Validate | docker-compose-validation |
Validate docker-compose.yaml structure |
| Integration | integration-test |
Start core services, verify health across Kafka, PostgreSQL, MySQL, Redis |
| Gate | pipeline-complete |
Aggregates all job statuses for branch protection enforcement |
fluxpipe/
├── airflow/
│ ├── Dockerfile
│ └── dags/
│ ├── batch_ingestion_dag.py # Daily batch ETL
│ ├── streaming_monitoring_dag.py # Kafka health monitoring
│ └── warehouse_transform_dag.py # Hourly warehouse ETL
├── spark/
│ ├── spark_batch_job.py # MinIO → transform → PostgreSQL
│ └── spark_streaming_job.py # Kafka consumer + anomaly detection
├── kafka/
│ └── producer.py # Sensor data generator
├── storage/
│ ├── aws_s3_influxdb.py
│ ├── hadoop_batch_processing.py
│ ├── mongodb_streaming.py
│ └── redis_integration.py
├── great_expectations/
│ └── expectations/raw_data_validation.py
├── governance/
│ └── atlas_stub.py # Apache Atlas lineage registration
├── ml/
│ ├── mlflow_tracking.py
│ └── feature_store_stub.py
├── monitoring/
│ ├── prometheus.yml
│ └── grafana-deployment-dashboards.json
├── sample_dotnet_backend/
│ ├── Dockerfile # Multi-stage .NET 8 build
│ └── src/DataPipelineApi/
│ ├── Controllers/ # 8 controllers
│ ├── Services/ # 11 services with interfaces
│ ├── Models/ # BatchRequest, StreamingRequest DTOs
│ ├── Options/ # 8 validated config classes
│ └── HealthChecks/ # 6 dependency health checks
├── snowflake/
│ ├── snowflake_connector.py
│ └── init_warehouse.sql # Star schema DDL + tasks + grants
├── helm/fluxpipe/
│ ├── Chart.yaml
│ ├── values.yaml
│ ├── values-aws.yaml
│ ├── values-gcp.yaml
│ ├── values-azure.yaml
│ ├── values-onprem.yaml
│ └── templates/ # 8 Kubernetes templates
├── kubernetes/ # Argo Rollouts, ingress, service monitors
├── terraform/ # VPC, EKS, RDS, S3, IAM, security groups
├── scripts/
│ ├── init_db.sql
│ ├── init_warehouse.sql
│ ├── deploy.sh # Universal deploy script
│ ├── deploy-blue-green.sh
│ ├── deploy-canary.sh
│ └── setup-advanced-deployments.sh
├── tests/ # 35 tests across 6 files
├── docker-compose.yaml # 20-service stack
├── docker-compose.ci.yaml
├── .env.example
├── Makefile
├── ARCHITECTURE.md
├── QUICK_START.md
└── DEPLOYMENT_STRATEGIES.md
All runtime configuration is managed through environment variables defined in .env.example. Copy the file and edit the relevant sections before starting the stack.
cp .env.example .env| Section | Key Variables |
|---|---|
| PostgreSQL | POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD |
| MySQL | MYSQL_DATABASE, MYSQL_USER, MYSQL_PASSWORD, MYSQL_ROOT_PASSWORD |
| Kafka | KAFKA_BROKER, KAFKA_TOPIC, KAFKA_ACKS_MODE |
| Spark | SPARK_MASTER_URL, SPARK_DRIVER_MEMORY, SPARK_EXECUTOR_MEMORY |
| Airflow | AIRFLOW__CORE__EXECUTOR, AIRFLOW_ADMIN_USER, AIRFLOW_ADMIN_PASSWORD |
| MinIO | MINIO_ROOT_USER, MINIO_ROOT_PASSWORD, MINIO_BUCKET_RAW, MINIO_BUCKET_PROCESSED |
| Grafana | GRAFANA_ADMIN_USER, GRAFANA_ADMIN_PASS |
| MLflow | MLFLOW_TRACKING_URI |
| Redis | REDIS_HOST, REDIS_PORT |
| MongoDB | MONGODB_URI, MONGODB_DB |
| InfluxDB | INFLUXDB_URL, INFLUXDB_TOKEN, INFLUXDB_ORG, INFLUXDB_BUCKET |
| Snowflake | SNOWFLAKE_ACCOUNT, SNOWFLAKE_USER, SNOWFLAKE_PASSWORD, SNOWFLAKE_WAREHOUSE, SNOWFLAKE_DATABASE |
| Governance | ATLAS_API_URL, ATLAS_USERNAME, ATLAS_PASSWORD |
PostgreSQL serves as the warehouse by default. To switch to Snowflake:
# 1. Set credentials in .env
SNOWFLAKE_ACCOUNT=your_account.us-east-1
SNOWFLAKE_USER=your_user
SNOWFLAKE_PASSWORD=your_password
# 2. Initialize the schema
snowsql -a $SNOWFLAKE_ACCOUNT -u $SNOWFLAKE_USER -f snowflake/init_warehouse.sql
# 3. The warehouse_transform_dag will automatically route to Snowflake when configured- Fork the repository
- Create a feature branch:
git checkout -b feature/your-feature - Commit your changes:
git commit -m 'Add your feature' - Push to the branch:
git push origin feature/your-feature - Open a Pull Request
This project is licensed under the MIT License.
For questions, feedback, or collaboration, reach out via GitHub.