We want a repeatable and reliable process for data ingestion. The options we had were: - Kubeflow (YAML based) - Cloud Composer ( use Apache Airflow, Python-based) - Apache Beam (Python Based) We decided to have ArgoCD as it is simple to define the tasks and reference with a simple DSL. We will need - [x] A GKE to be deployed to GCP - [ ] ArgoCD should be deployed to this GKE - [ ] At least one container to call from the workflow
We want a repeatable and reliable process for data ingestion. The options we had were:
Kubeflow (YAML based)
Cloud Composer ( use Apache Airflow, Python-based)
Apache Beam (Python Based)
We decided to have ArgoCD as it is simple to define the tasks and reference with a simple DSL.
We will need
A GKE to be deployed to GCP
ArgoCD should be deployed to this GKE
At least one container to call from the workflow