[Bug] SparkProcessBuilder fails to load spark.master from spark-defaults.conf

[wForget]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

The Spark engine fails to start because Kyuubi selects the wrong cluster manager, even though spark.master is configured in spark-defaults.conf.

org.apache.kyuubi.engine.spark.SparkProcessBuilder#defaultsConf is always empty. As a result, SparkProcessBuilder cannot load spark.master from spark-defaults.conf.

Since clusterManager() obtains the cluster manager through getSparkOption(MASTER_KEY), the missing configuration causes Kyuubi to use the wrong cluster manager when managing the Spark engine application. The engine application is then reported as NOT_FOUND, and engine startup fails.

error log:

2026-08-06 18:07:28.615 ERROR LaunchEngine: Error operating LaunchEngine:
The engine application has been terminated. Please check the engine log.
ApplicationInfo: (
name -> null,
id -> null,
error -> null,
state -> NOT_FOUND,
url -> null
)

org.apache.kyuubi.KyuubiSQLException:
The engine application has been terminated. Please check the engine log.
ApplicationInfo: (
name -> null,
id -> null,
error -> null,
state -> NOT_FOUND,
url -> null
)

        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$11(EngineRef.scala:340)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$11$adapted(EngineRef.scala:329)
        at scala.Option.foreach(Option.scala:437)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$10(EngineRef.scala:329)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$10$adapted(EngineRef.scala:318)
        at scala.Option.foreach(Option.scala:437)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$create$1(EngineRef.scala:318)
        at org.apache.kyuubi.ha.client.zookeeper.ZookeeperDiscoveryClient.tryWithLock(ZookeeperDiscoveryClient.scala:166)
        at org.apache.kyuubi.engine.EngineRef.tryWithLock(EngineRef.scala:221)
        at org.apache.kyuubi.engine.EngineRef.create(EngineRef.scala:226)
        at org.apache.kyuubi.engine.EngineRef.$anonfun$getOrCreate$1(EngineRef.scala:377)
        at scala.Option.getOrElse(Option.scala:201)
        at org.apache.kyuubi.engine.EngineRef.getOrCreate(EngineRef.scala:377)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2(KyuubiSessionImpl.scala:160)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$2$adapted(KyuubiSessionImpl.scala:134)
        at org.apache.kyuubi.ha.client.DiscoveryClientProvider$.withDiscoveryClient(DiscoveryClientProvider.scala:36)
        at org.apache.kyuubi.session.KyuubiSessionImpl.$anonfun$openEngineSession$1(KyuubiSessionImpl.scala:134)
        at org.apache.kyuubi.session.KyuubiSession.handleSessionException(KyuubiSession.scala:49)
        at org.apache.kyuubi.session.KyuubiSessionImpl.openEngineSession(KyuubiSessionImpl.scala:134)
        at org.apache.kyuubi.operation.LaunchEngine.$anonfun$runInternal$1(LaunchEngine.scala:60)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

The issue may have been introduced by #6008 due to accessing subclass state during initialization.

As shown in the screenshot, sparkHome is still null when SparkProcessBuilder#defaultsConf is evaluated because it has not been initialized yet. Consequently, the path to spark-defaults.conf is resolved incorrectly, causing defaultsConf to be empty and preventing spark.master from being loaded.

Affects Version(s)

master

Kyuubi Server Log Output

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[wForget]

This issue may have been introduced by #6008, we should not call the following code in ProcBuilder.$init$:

https://github.com/zhaohehuhu/incubator-kyuubi/blob/00c208a26e17668d287a4c995d5ec274a4001f4b/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/ProcBuilder.scala#L172-L175