[Bug][Spark] Spark SQL engine shutdown fails when no active Spark session is available

[wForget]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

I encountered an unexpected exception:

26/08/11 23:53:47 ERROR AsyncEventQueue: Listener SparkSQLEngineListener threw an exception
org.apache.spark.SparkException: [INTERNAL_ERROR] No active or default Spark session found SQLSTATE: XX000
	at org.apache.spark.SparkException$.internalError(SparkException.scala:104)
	at org.apache.spark.SparkException$.internalError(SparkException.scala:108)
	at org.apache.spark.sql.SparkSessionCompanion.$anonfun$active$2(SparkSession.scala:1037)
	at scala.Option.getOrElse(Option.scala:201)
	at org.apache.spark.sql.SparkSessionCompanion.$anonfun$active$1(SparkSession.scala:1037)
	at scala.Option.getOrElse(Option.scala:201)
	at org.apache.spark.sql.SparkSessionCompanion.active(SparkSession.scala:1036)
	at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.globalSparkContext(KyuubiSparkUtil.scala:41)
	at org.apache.kyuubi.engine.spark.KyuubiSparkUtil$.engineId(KyuubiSparkUtil.scala:58)
	at org.apache.kyuubi.engine.spark.session.SparkSQLSessionManager.getEngineResultSavePath(SparkSQLSessionManager.scala:218)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineSavePath$lzycompute(SparkSQLEngine.scala:62)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.engineSavePath(SparkSQLEngine.scala:61)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.$anonfun$stop$3(SparkSQLEngine.scala:119)
	at org.apache.kyuubi.Logging.error(Logging.scala:88)
	at org.apache.kyuubi.Logging.error$(Logging.scala:86)
	at org.apache.kyuubi.service.AbstractService.error(AbstractService.scala:23)
	at org.apache.kyuubi.engine.spark.SparkSQLEngine.stop(SparkSQLEngine.scala:119)
	at org.apache.spark.kyuubi.SparkSQLEngineListener.onApplicationEnd(SparkSQLEngineListener.scala:61)
	at org.apache.spark.scheduler.SparkListenerBus.doPostEvent(SparkListenerBus.scala:57)
	at org.apache.spark.scheduler.SparkListenerBus.doPostEvent$(SparkListenerBus.scala:28)
	at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:38)
	at org.apache.spark.scheduler.AsyncEventQueue.doPostEvent(AsyncEventQueue.scala:38)
	at org.apache.spark.util.ListenerBus.postToAll(ListenerBus.scala:118)
	at org.apache.spark.util.ListenerBus.postToAll$(ListenerBus.scala:102)
	at org.apache.spark.scheduler.AsyncEventQueue.super$postToAll(AsyncEventQueue.scala:106)
	at org.apache.spark.scheduler.AsyncEventQueue.$anonfun$dispatch$1(AsyncEventQueue.scala:106)
	at scala.runtime.java8.JFunction0$mcJ$sp.apply(JFunction0$mcJ$sp.scala:17)
	at scala.util.DynamicVariable.withValue(DynamicVariable.scala:59)
	at org.apache.spark.scheduler.AsyncEventQueue.org$apache$spark$scheduler$AsyncEventQueue$$dispatch(AsyncEventQueue.scala:101)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.$anonfun$run$1(AsyncEventQueue.scala:97)
	at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1293)
	at org.apache.spark.scheduler.AsyncEventQueue$$anon$2.run(AsyncEventQueue.scala:97)

Affects Version(s)

1.12.0

Kyuubi Server Log Output

Kyuubi Engine Log Output

Kyuubi Server Configurations

Kyuubi Engine Configurations

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.