[FEATURE] Coordinate terminated Spark Driver Pod cleanup across Kyuubi servers

[ruanwenjun]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched the existing issues and found no issue covering coordinated cleanup across multiple Kyuubi servers.

Describe the feature

When multiple Kyuubi server instances monitor the same Kubernetes context and namespace, each instance starts a Pod informer and maintains its own terminated-application cleanup cache.

After a Spark Driver Pod reaches a terminal state, every server instance that observed the event independently expires its local cache entry and calls the Kubernetes DELETE API:

https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala#L145-L210

This has two side effects:

1. For one terminated Driver Pod, the API server may receive one DELETE request from every Kyuubi server instance. Only the first request deletes the Pod; later requests normally observe that it is already absent.
2. An empty result from Fabric8 DELETE is currently logged as "Failed to delete pod", although a named-resource HTTP 404 is converted by Fabric8 into an empty result. Real authorization, throttling, network, or server failures are therefore difficult to distinguish from an idempotent no-op.

The DELETE event is also passed to markApplicationTerminated:

https://github.com/apache/kyuubi/blob/master/kyuubi-server/src/main/scala/org/apache/kyuubi/engine/KubernetesApplicationOperation.scala#L400-L434

The cleanup state does not record that the Pod is already deleting or absent. An existing cleanup trigger is not canceled, and a trigger that has already expired may be created again, causing another redundant DELETE attempt later.

The cleanup responsibility should be coordinated independently from application-state observation.

Motivation

The number of mutating Kubernetes API requests for terminated Driver Pods currently grows with the number of Kyuubi server replicas. The resulting warning logs are also not actionable because they conflate an already-absent Pod with a real DELETE failure.

A coordinated cleaner would reduce DELETE requests and preserve useful failure diagnostics without changing how individual Kyuubi servers monitor applications.

This is different from #7293, which is about Kubernetes clients and informers not being initialized in some deployments.

Describe the solution

A possible design is:

1. Elect one cleanup leader for each KubernetesInfo(context, namespace) through the existing Kyuubi HA backend (ZooKeeper or etcd).
2. Keep Pod informers on all Kyuubi servers for application-state tracking, audit logging, metadata persistence, cancellation handling, and local getApplicationInfoByTag calls. Leadership gates only the cleanup responsibility.
3. Only the cleanup leader schedules or executes deletion of terminated Driver Pods for its context and namespace.
4. When leadership is acquired, list and reconcile existing terminated Driver Pods so that cleanup continues after the previous leader fails or restarts.
5. When leadership is lost, stop issuing cleanup DELETE requests.
6. Treat a Pod with metadata.deletionTimestamp, or an observed DELETE event, as already satisfying cleanup and do not enqueue it again.
7. Log an accepted DELETE request as such, treat an empty/not-found result as an idempotent no-op, and reserve warning/error logs for real exceptions.