Workflows in existing clusters

[gdonoso94]

Hi all,

I've been trying for a while and I believe it's not supported yet but I'm asking anyway... Is there any way to deploy a workflow in an existing cluster? So far I've managed to deploy it in a new_cluster like this:

custom:
  basic-cluster-props: &basic-cluster-props
    spark_version: "12.2.x-scala2.12"

  basic-static-cluster: &basic-static-cluster
    <<: *basic-cluster-props
    num_workers: 2
    node_type_id: "i3.xlarge"

environments:
  default:
    workflows:
      #######################################################################################
      #   Example workflow for integration tests                                            #
      #######################################################################################
      - name: "box-sample-tests"
        tasks:
          - task_key: "main"
            <<: *basic-static-cluster
            spark_python_task:
              python_file: "file://tests/entrypoint.py"
              # this call supports all standard pytest arguments
              parameters: ["file:fuse://tests/integration", "--cov=box"]

And after that, change the cluster replacing <<: *basic-static-cluster for existing_cluster_id: "<id>".

If I try to deploy this workflow for the first time using the existing_cluster_id, I get the following error

ValidationError: 1 validation error for Deployment
workflows -> 2 -> Workflow -> job_clusters -> 0 -> new_cluster
  field required (type=value_error.missing)

Am I missing something?

Thanks

P.S.: I know this is not the recommended way to deploy jobs

[sidharthbolar]

Yes there is
Above the spark_python_task:
Add the following

existing_cluster_id : ACTUAL_CLUSTER_ID

[gdonoso94]

Thanks!

My problem was that I was trying to reuse the job_cluster_key