Skip to content

fix: Correctly propage user arguments to Spark connect server#696

Open
sbernauer wants to merge 3 commits into
mainfrom
fix/connect-user-argument-list
Open

fix: Correctly propage user arguments to Spark connect server#696
sbernauer wants to merge 3 commits into
mainfrom
fix/connect-user-argument-list

Conversation

@sbernauer

@sbernauer sbernauer commented Jun 10, 2026

Copy link
Copy Markdown
Member

Description

A customer asked me how to do Iceberg with a SparkConnect server. So I tried and noticed this bug.
Added a kuttl test as well.

From

  - args:                                                                                                                                                                                                          
    - |-                                                                                                                                                                                                           
      containerdebug --output=/stackable/log/containerdebug-state.json --loop &                                                                                                                                    
                                                                                                                                                                                                                   
      cp /stackable/spark/conf/spark-defaults.conf /tmp/spark.properties                                                                                                                                           
      config-utils template /tmp/spark.properties                                                                                                                                                                  
                                                                                                                                                                                                                   
      /stackable/spark/sbin/start-connect-server.sh \                                                                                                                                                              
      --deploy-mode client \                                                                                                                                                                                       
      --master k8s://https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT_HTTPS} \                                                                                                                         
      --properties-file /tmp/spark.properties                                                                                                                                                                      
       --package org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0 

to

  - args:                                                                                                                                                                                                          
    - |-                                                                                                                                                                                                           
      containerdebug --output=/stackable/log/containerdebug-state.json --loop &                                                                                                                                    
                                                                                                                                                                                                                   
      cp /stackable/spark/conf/spark-defaults.conf /tmp/spark.properties                                                                                                                                           
      config-utils template /tmp/spark.properties                                                                                                                                                                  
                                                                                                                                                                                                                   
      /stackable/spark/sbin/start-connect-server.sh \                                                                                                                                                              
      --deploy-mode client \                                                                                                                                                                                       
      --master k8s://https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT_HTTPS} \                                                                                                                         
      --properties-file /tmp/spark.properties \                                                                                                                                                                    
      --packages org.apache.iceberg:iceberg-spark-runtime-4.1_2.13:1.11.0

Definition of Done Checklist

  • Not all of these items are applicable to all PRs, the author should update this template to only leave the boxes in that are relevant
  • Please make sure all these things are done and tick the boxes

Author

  • Changes are OpenShift compatible
  • CRD changes approved
  • CRD documentation for all fields, following the style guide.
  • Helm chart can be installed and deployed operator works
  • Integration tests passed (for non trivial changes)
  • Changes need to be "offline" compatible
  • Links to generated (nightly) docs added
  • Release note snippet added

Reviewer

  • Code contains useful comments
  • Code contains useful logging statements
  • (Integration-)Test cases added
  • Documentation added or updated. Follows the style guide.
  • Changelog updated
  • Cargo.toml only contains references to git tags (not specific commits or branches)

Acceptance

  • Feature Tracker has been updated
  • Proper release label has been added
  • Links to generated (nightly) docs added
  • Release note snippet added
  • Add type/deprecation label & add to the deprecation schedule
  • Add type/experimental label & add to the experimental features tracker

@sbernauer sbernauer marked this pull request as ready for review June 10, 2026 12:27
@sbernauer sbernauer requested a review from razvan June 10, 2026 12:27
@sbernauer sbernauer self-assigned this Jun 10, 2026
@sbernauer sbernauer moved this to Development: Waiting for Review in Stackable Engineering Jun 10, 2026
@razvan razvan moved this from Development: Waiting for Review to Development: In Review in Stackable Engineering Jun 10, 2026

@razvan razvan left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

Status: Development: In Review

Development

Successfully merging this pull request may close these issues.

3 participants