[FEATURE][SERVER] Support Spark Connect protocol — make Kyuubi a multi-tenant gRPC gateway for Spark Connect clients

[turboFei]

Summary

Spark Connect (GA in Apache Spark 3.4) defines a language-agnostic gRPC protocol that decouples clients from the Spark driver. Today Kyuubi only exposes Thrift/JDBC/REST surfaces; PySpark users connecting via spark.remote("sc://...") or any non-JVM thin client (Go, Rust) cannot benefit from Kyuubi's multi-tenancy, session pooling, and access control.

Supporting Spark Connect transforms Kyuubi into a unified gateway for both legacy JDBC workloads and modern DataFrame/Spark Connect clients, with no changes required on the Spark engine side.

Motivation

Today	With this feature
PySpark thin clients bypass Kyuubi entirely	All Spark Connect clients authenticate through Kyuubi
Non-JVM clients (Go, Rust) have no path to Kyuubi	Any Spark Connect-compatible client works
JDBC is the only managed multi-tenant surface	gRPC + JDBC both benefit from Kyuubi's session pooling and access control

Proposed Architecture

flowchart TD
    subgraph Clients
        A1["PySpark\nspark.remote('sc://...')"]
        A2["Go / Rust\nSpark Connect client"]
    end

    subgraph KyuubiServer ["Kyuubi Server (kyuubi-spark-connect module)"]
        B["NettyGrpcServer\n:15002"]
        C["AuthInterceptor\n(Bearer token → KyuubiUser)"]
        D["SparkConnectFrontendService\n(SparkConnectServiceGrpc)"]
        E["SparkConnectSessionManager\n(session lifecycle, user isolation)"]
        F["SparkConnectEngineProxy\n(gRPC channel to engine)"]
        G["ServiceDiscovery\n(ZooKeeper / Kubernetes)"]
    end

    subgraph EngineAlice ["Spark Engine – user alice"]
        H1["SparkConnect gRPC\n(embedded, auto-port)"]
        H2["SparkSession"]
    end

    subgraph EngineBob ["Spark Engine – user bob"]
        I1["SparkConnect gRPC\n(embedded, auto-port)"]
        I2["SparkSession"]
    end

    A1 -- "gRPC (Spark Connect proto)" --> B
    A2 -- "gRPC (Spark Connect proto)" --> B
    B --> C
    C -- "KyuubiUser" --> D
    D --> E
    E -- "open / reuse session" --> F
    F -- "discover sparkConnectPort" --> G
    F -- "proxy protobuf stream\n(alice)" --> H1
    F -- "proxy protobuf stream\n(bob)" --> I1
    H1 --- H2
    I1 --- I2

Loading

Key design choice — proxy, not embed: Kyuubi forwards Spark Connect protobuf streams to a Spark engine running in --spark-connect mode. Kyuubi does not embed a SparkSession, preserving the fundamental server/engine module boundary.

Proposed Changes

• New module kyuubi-spark-connect (profile-gated via -Pspark-connect) containing the gRPC server, protobuf stubs, and session bridge.
• New gRPC listener implementing SparkConnectService proto on port 15002 (configurable), alongside the existing Thrift frontend.
• Session management extension: Spark Connect sessions map onto Kyuubi's existing SessionManager / OperationManager with a new SparkConnectSession type; per-user engine limits and idle-timeout apply unchanged.
• Engine-side proxy: transparent byte-level forwarding of ExecutePlan, AnalyzePlan, Config, Interrupt, and ReattachExecute RPCs to the per-user Spark engine.
• Engine port discovery: engine registers its Spark Connect gRPC port in ZooKeeper (node attribute sparkConnectPort) or Kubernetes (pod label kyuubi.apache.org/spark-connect-port).
• Auth: AuthInterceptor validates bearer tokens via Kyuubi's existing AuthenticationProvider chain; server-authenticated identity overrides any client-supplied UserContext.user_name.
• New config namespace kyuubi.frontend.spark.connect.* (enabled, bind host/port, max message size, TLS options).
• Feature off by default (kyuubi.frontend.spark.connect.enabled=false); fully additive and non-breaking.

New Dependencies

Artifact	License	Scope
grpc-netty-shaded	Apache 2.0	-Pspark-connect only
protobuf-java	BSD-3	-Pspark-connect only
spark-connect-common (Spark 3.5+)	Apache 2.0	-Pspark-connect only

Open Questions for Community Discussion

1. Should the initial scope include ReattachExecute (required for long-running streaming queries), or defer to a follow-up?
2. Should AddArtifacts (client-streaming file/JAR upload) be in scope for v1?
3. Should kyuubi-spark-connect live as a top-level module or under extensions/server/?
4. How should the Spark Connect auth token model interact with Ranger/HMS auth plugins — does the bearer token carry the same identity used for SQL authorization?
5. Which Spark Connect proto version to target first? (Spark 3.5 is the most widely deployed LTS, but Spark 4.0 changed the proto significantly.)

Related

• Planning artifacts (proposal, design, specs, tasks): [KYUUBI][PROPOSAL] Add Spark Connect gRPC server support to Kyuubi #7510
• Apache Spark Connect documentation: https://spark.apache.org/docs/latest/spark-connect-overview.html
• SparkConnectService proto: connector/connect/common/src/main/protobuf/spark/connect/base.proto in the Spark repo

[asafb2k]

Need this feature as well

[alexandrefimov]

Hi @turboFei, I'd like to work on this. I read #7510 and the older #6642.

There are three things I'd like to settle before touching code:

1. Should Kyuubi only authenticate and pin a Connect session to an engine, while the client and engine keep ownership of session_id, server_side_session_id, operation_id, and reattach state? [KYUUBI][PROPOSAL] Add Spark Connect gRPC server support to Kyuubi #7510 also maps this state into SessionManager and OperationManager, which makes Kyuubi part of the Connect state machine.

2. Should plan-level authorization stay in the Spark engine, or should Kyuubi inspect and authorize DataFrame plans before forwarding them?

3. Should [KYUUBI][PROPOSAL] Add Spark Connect gRPC server support to Kyuubi #7510 be used as the design baseline, or only as background? Was it closed for any reason that should affect new work?

For v1, my preference is a thin proxy: one exact client/engine protocol line, no ExecutePlan replay after an ambiguous disconnect, and no reattach to another engine. Kyuubi would own authentication, routing, limits, and audit; Spark would own plan execution and operation state.

If that matches the intended boundary, I can update the design from #7510 and split the work into reviewable PRs.

[turboFei]

Hi @alexandrefimov,

Please treat #7510 as background material only, rather than an agreed design baseline. Feel free to lead the design and use #7510 and #6642 as references where useful.

There is now an upstream Spark proposal that we should evaluate before implementing a Kyuubi-specific gateway: SPARK-58455: Spark Connect Gateway — Session Affinity, Auth, and Multi-Tenancy for Spark Connect Servers. The full SPIP document proposes an official standalone, wire-compatible gateway covering session affinity, operation reattach routing, authentication, multi-tenancy, rate limiting, auditing, backend discovery, and HA.

This overlaps substantially with the thin-proxy responsibilities proposed in #7511. One important boundary is that the upstream gateway explicitly does not manage Spark driver lifecycle: it consumes an existing pool of Spark Connect servers and delegates provisioning and scaling to an external system.

Therefore, I suggest that the design first evaluate how Kyuubi should integrate with the upstream gateway proposal. A possible boundary is:

• The Spark Connect Gateway owns protocol-aware routing, session affinity, authentication, and reattach routing.
• Spark Connect servers own SparkSession, operation state, plan execution, and plan-level authorization.
• Kyuubi owns engine provisioning and lifecycle, tenant resource policies, engine discovery, and integration with the rest of the Kyuubi ecosystem.

If a native Kyuubi gateway is still needed, the design should explain what it provides beyond SPARK-58455 and how we can avoid maintaining duplicate implementations of Spark Connect routing and protocol semantics.

Since SPARK-58455 is still an open SPIP, the final upstream direction is not settled yet. Please feel free to lead this investigation and propose the Kyuubi-side design. It would also be useful to participate in the upstream discussion so the two designs can evolve together.

[alexandrefimov]

Thanks @turboFei. I went through SPARK-58455 and the reference implementation. I agree that Kyuubi should not build another Connect-aware proxy.

I found one integration gap that matters for Kyuubi’s default USER share level. The gateway includes user_id in the affinity key, but a new session is currently placed round-robin within a tenant pool. If Kyuubi publishes several per-user engines into the same pool, there is nothing yet that prevents Alice’s first session from being sent to Bob’s engine. The routing interface already receives the full session key, so backend owner metadata plus a user-aware selection policy may be enough. I’ll raise that upstream.

On the Kyuubi side, the remaining scope looks like:

• optionally start Spark Connect in a Kyuubi-launched Spark engine and advertise its endpoint through the existing discovery metadata;
• expose the live engine pool to the gateway, including YARN deployments;
• coordinate idle shutdown and drain with Connect activity;
• provide the backend authentication token when the engine is launched.

I would keep routing, affinity and Connect authentication in the upstream gateway, and execution and authorization in Spark. Per-user engines should preserve Kyuubi’s existing execution identity, but I would validate that with Ranger end to end rather than assume it works automatically.

For the first Kyuubi slice, I’d target one Spark 4.x line, disabled by default: start Connect alongside Thrift and advertise the endpoint. Does that look like the right first slice, especially if user-aware backend placement is handled in the upstream gateway?

[turboFei]

Yes, this looks like the right direction and a reasonable first slice.

The user-aware placement gap you identified is important for Kyuubi’s USER share level. Raising it in SPARK-58455 makes sense. Backend ownership should be matched against the identity verified by the gateway, rather than a client-supplied user field.

For the initial Kyuubi slice, I suggest keeping the scope narrow:

• target one Spark 4.x profile initially and keep the feature disabled by default;
• start Spark Connect inside the Spark SQL engine, alongside the existing Thrift frontend, without introducing Spark runtime dependencies into kyuubi-server;
• advertise the bound Connect endpoint and the minimum required backend metadata through the existing engine discovery mechanism;
• start with the USER share level and document other share levels as follow-up work;
• configure backend authentication when launching the engine, but do not publish authentication secrets in discovery metadata.

One lifecycle detail should be included in the first slice: existing engine idle and graceful-shutdown logic tracks Thrift sessions and operations. A Connect session must keep the engine alive, and draining an engine must remove its Connect endpoint before waiting for active Connect work to finish. Otherwise an engine could be considered idle and terminated while it still has an active Connect session.

YARN gateway-pool integration can be a separate follow-up after endpoint advertisement is established, since it likely needs a Kyuubi-aware discovery integration on the upstream gateway side.

Validating the execution identity and Ranger behavior end to end is also the right approach. We should not assume that identity propagation through the gateway automatically preserves the existing authorization semantics.

Please go ahead with a design for this first slice before starting the implementation.

[alexandrefimov]

Spark Connect in the Kyuubi Spark engine — design for the first slice

Spark Connect does not use user_id for anything except the session key: a plan is executed and authorized as the driver's own user. That single fact shapes most of what follows, so the evidence first.

Static checks on 4.0.3, 4.1.2 and 4.2.0: the org.apache.spark.sql.connect package references neither UserGroupInformation nor CurrentUserContext, and ExecuteThreadRunner sets only callSite.short / callSite.long and job tags. Two runtime checks, both on 4.0.3: a driver started with --proxy-user testalice serving a client connected as user_id=bob returns testalice from current_user() and session_user() — SPARK_USER was unset, so that value comes from UserGroupInformation.getCurrentUser on the planning thread; and with kyuubi-spark-authz loaded into the same driver, a CREATE TABLE from the bob client is denied with Permission denied: user [testalice] does not have [create] privilege on [default/poc_authz_probe].

At USER share level with kyuubi.engine.doAs.enabled=true (the default), the driver already runs as the end user, so authorization identity matches the Thrift path — by deployment topology rather than by propagation. Three limitations remain, and I would rather document them than paper over them:

• with doAs disabled, the Thrift path still authorizes as the session user through the kyuubi.session.user local property set in SparkOperation.withLocalProperties (SparkOperation.scala:164) and read first by AuthZUtils.getAuthzUgi; Connect sets no such property, so it would authorize as the server user — a regression relative to Thrift at the same share level;
• Connect operations bypass withLocalProperties entirely (SparkOperation.scala:158-194), so the scheduler pool — tenant resource policies — the statement id and the session-user signature are absent for Connect queries;
• with kyuubi.session.user.sign.enabled, AuthZUtils.getAuthzUgi calls verifyKyuubiSessionUser with a null user and throws Invalid user identifier [null] (AuthZUtils.scala:51-53, :123-133). It fails closed, but the feature is unusable in those deployments.

Scope

As you outlined it. Engines are still provisioned through the existing session path; this slice starts Connect inside the engine and advertises the endpoint. Routing a Connect client to an engine is gateway-side work and out of scope, together with YARN pool integration, share levels other than USER, and token rotation.

The decisions in one place, before the detail: Connect is started from configuration and its state read back by reflection, so nothing Connect-specific is compiled into the engine module; the bound endpoint becomes a new discovery attribute; the engine's active-session count starts including Connect sessions, which is what makes the existing termination paths see them, and drain deregisters before it waits. A backend token is required whenever Connect is enabled and is kept off the command line, and any share level other than USER fails at startup.

Starting Connect

spark-connect_2.13-<version>.jar ships in the standard Spark 4.x distribution and org.apache.spark.sql.connect.SparkConnectPlugin implements SparkPlugin, so starting the service is configuration only. SparkProcessBuilder appends the plugin class to spark.plugins — appends, since users set it for Delta and others — along with the Connect settings, and no Spark Connect code or dependency enters kyuubi-server.

Reading state back — the bound port, the live sessions — does need the Connect classes, and that has to go through reflection: SparkConnectService's Scala signatures reference io.grpc, so a compile-time reference fails with Symbol 'term io.grpc' is missing from the classpath unless grpc is added to the engine module's compile classpath. Reflection also keeps the default build inert, since the module compiles against the active Spark profile (spark-3.5 by default) and has no Spark-version-specific source directories.

The reflection targets are identical on 4.0.3, 4.1.2 and 4.2.0 — SparkConnectService$.localPort() / hostAddress() / started(), sessionManager().listActiveSessions(), listActiveExecutions(), stop(Option[Long], Option[TimeUnit]) — so I expect one implementation to cover the 4.x profiles in the tree. A capability check at startup fails the engine with an explicit message if they are missing, rather than degrading silently.

Configuration entry: kyuubi.engine.spark.connect.enabled, default false, version("1.12.0"), immutable and admin-controlled. The engine key (EngineRef.engineSpace, EngineRef.scala:187-194) does not include it, so a per-session value would let one user's sessions disagree about whether their engine has Connect — the second session would silently reuse an engine with no endpoint. Immutable keeps the flag uniform per deployment; a client still tells the two apart by whether the endpoint attribute is present in discovery, since changing the flag does not retrofit a running engine. settings.md is regenerated in the same PR as the entry.

Discovery

SparkTBinaryFrontendService.attributes (externals/kyuubi-spark-sql-engine/.../SparkTBinaryFrontendService.scala:99-108) already publishes kyuubi.engine.id and kyuubi.engine.url. This slice adds the bound endpoint as kyuubi.engine.connect.url — a new KyuubiReservedKeys constant next to KYUUBI_ENGINE_URL (KyuubiReservedKeys.scala:34-36) — with host:port as the value, read back after the plugin binds so that spark.connect.grpc.binding.port=0 works.

The value shape is not a style choice. Attributes are serialised into a single discovery node-name element, k=v joined with ; (kyuubi-ha/.../ZookeeperDiscoveryClient.scala:361-366, same in EtcdDiscoveryClient.scala:332-338), so a value can contain neither ; nor / — = is safe, since decoding splits with a limit of two (ZookeeperDiscoveryClient.scala:206-208). That is why kyuubi.engine.url is already stored with its scheme stripped (SparkTBinaryFrontendService.scala:105-108), and why the Connect endpoint is published without an sc:// prefix. The same encoding means the endpoint cannot be deregistered independently of the Thrift node — removing one removes both, which is what drain below relies on. Attributes also already surface through GET /api/v1/admin/engine (AdminResource.scala:377), so operators see the endpoint with no new REST work.

Ownership is already implied by the discovery path for USER share level (/serverSpace_version_USER_engineType/<user>[/<subdomain>]), so this slice publishes only the endpoint. Whether the attribute set should also carry an explicit owner label is a question I would rather settle together with the gateway-side placement work. No secrets are published here.

Lifecycle

Engine termination is driven by SessionManager.getActiveUserSessionCount (kyuubi-common/.../session/SessionManager.scala:167) through four call sites: the idle checker (SessionManager.scala:346-373), the max-lifetime checker (SparkSQLEngine.scala:165-214), gracefulStop (SparkSQLEngine.scala:123-138) and ServiceDiscovery.stopGracefully (kyuubi-ha/.../ServiceDiscovery.scala:68-74, reached on discovery connection loss or node deletion — the kyuubi-ctl path). startFastFailChecker uses the same counter but only runs at CONNECTION share level (SparkSQLEngine.scala:88-91), so it is out of reach for this slice.

A Connect session is invisible to all of them. Connect exposes what is needed: listActiveSessions() returns per-session records carrying userId and lastAccessTimeMs, and listActiveExecutions() returns either the active executions or — when none is active — a System.nanoTime() reading of the last completion. That reading is monotonic and unrelated to wall clock, so it has to be converted against a nanoTime baseline before it can feed the idle timer, which compares against System.currentTimeMillis() (SessionManager.scala:355).

The plan:

SparkSQLSessionManager includes live Connect sessions in the active count, which reaches all four call sites without touching kyuubi-common. The cost is that getActiveUserSessionCount acquires a second meaning: it no longer agrees with allSessions(), and its other consumers — ServiceDiscovery.stopGracefully, the "current opening sessions" log line, the engine UI — inherit that. The alternative is a separate overridable predicate (hasActiveWork or similar) consulted next to the counter, which keeps the counter honest at the cost of touching every call site. See question 1.

Idle detection also needs a last-activity source. latestLogoutTime is readable, but it is advanced only inside SessionManager.closeSession (SessionManager.scala:124-131) and at checker start (:346-348), so a closing Connect session leaves it untouched and the idle timer is already expired when the last one goes away.

The max-lifetime path needs an explicit answer rather than the counter alone. After maxLifetime + gracefulPeriod it force-closes sessions drawn from allSessions() (SparkSQLEngine.scala:184-202); Connect sessions are not in handleToSession, so they would never be force-closed while the counter never reaches zero — an engine past its max lifetime with one idle-but-open Connect session would never terminate. Either the graceful period force-closes Connect sessions through SparkConnectService as well, or the slice relies on spark.connect.session.manager.defaultSessionTimeout and says so.

Drain: gracefulStop deregisters from discovery — which removes the Connect endpoint with the node, as above — then waits for Connect sessions and executions to finish, and only then calls SparkConnectService.stop(grace, unit). The order matters: that call starts with a gRPC shutdown() before awaiting termination, and afterwards a client is answered with UNAVAILABLE; since every Connect statement is a separate RPC, stopping the service first would cut sessions that are still draining.

Backend authentication

spark.connect.authenticate.token is present in the 4.0, 4.1 and 4.2 lines I checked; on 4.0.3 a client connecting without it is rejected with UNAUTHENTICATED. With Connect enabled the engine should refuse to start unless a token is configured — otherwise flipping the flag yields an open gRPC endpoint that accepts any user_id and executes with the engine owner's privileges, which is exactly the failure the section above describes. The check has to cover the runtime as well as the configuration: the key only exists from 4.0, while the Connect plugin itself is present on 3.5, so a capability check that probes only the reflection targets would pass on a 3.5 runtime with the token silently ignored. Gating on KyuubiSparkUtil.SPARK_ENGINE_RUNTIME_VERSION >= "4.0" (KyuubiSparkUtil.scala:114) closes that.

On delivery: configuration reaching the engine goes through argv — SparkProcessBuilder.commands expands conf.getEngineConf(EngineType.SPARK_SQL) into --conf k=v pairs (SparkProcessBuilder.scala:133-157) and Utils.redactCommandLineArgs (Utils.scala:329) only touches the logged copy — so a SPARK-audience token key would be readable in ps by any user on the host. The precedent for avoiding that is already in the tree: FlinkProcessBuilder.generateTokenFile (FlinkProcessBuilder.scala:281-297) writes a generated credential to a file and passes the path through an env var, cleaning up in close. I would follow that shape, with the token placed into _sparkConf before createSpark() (SparkSQLEngine.scala:328-334), since the plugin binds inside getOrCreate.

I am aware this argues against an existing pattern: engine credentials travel through the launch conf today (KyuubiSessionImpl.scala:137-139, read back at SparkSQLEngine.scala:328). The difference I would draw is that those are per-session Hadoop delegation tokens with their own expiry, while this one is the only thing standing between an unauthenticated client and an engine running with a specific user's privileges. Note also that the Spark UI and event logs are covered by default — the engine's default spark.redaction.regex already includes token (SparkSQLEngine.scala:273-276; an operator who overrides the pattern keeps that responsibility) — so the file buys exactly one thing, keeping the secret out of argv.

One token per engine rather than one per deployment, so that a leaked token costs a single engine rather than every engine sharing it. In cluster deployments the file still has to reach the driver host (staging dir, spark.files, or a Secret); which mechanism to use is open — see question 2.

Tests and cost

The tests that exercise a live session — keep-alive, drain and the token rejection — need a Spark Connect client on the test classpath. There is no spark-connect dependency in the tree today, so that is a new test-scoped dependency plus a runtime-version assume in the style of SparkOperationSuite.scala:200. It is the largest hidden cost in the slice and I would rather name it up front than discover it in review.

With the flag off, behaviour is unchanged; with it on, the Connect port binds and the endpoint is advertised, and the node carries no secret. An engine with a live Connect session is not reaped by the idle timeout, and drain removes the endpoint before waiting for active work. A client connecting directly without the token is rejected.

PRs

Tests ship with the feature in each PR, and settings.md is regenerated in the PR that adds the config entry.

1. start Connect: config entry, process builder, capability check, settings.md;
2. advertise the endpoint through discovery;
3. lifecycle: keep-alive, max-lifetime handling and drain;
4. backend token at engine launch;
5. deployment documentation and the share-level follow-up note.

Happy to merge 1 and 2 if you would rather review one self-contained slice.

Open questions

1. Connect activity in the lifecycle checkers: include Connect sessions in getActiveUserSessionCount (reaches all four call sites, but the counter stops agreeing with allSessions() for its other consumers), or add a separate overridable predicate next to it and update the call sites?
2. Secret delivery: follow FlinkProcessBuilder.generateTokenFile — generated file plus path in an env var — and if so, which distribution mechanism for cluster modes?
3. Fail at startup for non-USER share levels and for a missing token, as proposed, or warn and continue?
4. The 4.0, 4.1 and 4.2 jobs already run the engine suites (.github/workflows/master.yml:64-81), so I would gate the new tests with assume(SPARK_ENGINE_RUNTIME_VERSION >= "4.0") and let them run in all three — any reason to restrict that?

[alexandrefimov]

Kyuubi's engine already carries the piece that makes the upstream fix cover our case. SparkSQLEngine.scala:273-276 appends |zookeeper.auth.digest to spark.redaction.regex in _sparkConf, before the session is built (:333), so the pattern sits in sparkContext.conf — where the Config RPC patch in apache/spark#57892 (SPARK-58658, two approvals, not merged yet) reads it. spark.kyuubi.ha.zookeeper.auth.digest, spark.kyuubi.internal.security.secret.provider.simple.secret and spark.connect.authenticate.token all match that pattern, so on a Spark line carrying the fix the handler withholds them on a key match — that append is what makes it reach our keys, and nothing new is needed on our side. That is a reading of both sides' code: I have run the leak on 4.0.3 and 4.2.0, but not yet a Kyuubi engine against a patched Spark build. It also assumes a pattern covering secret and token; an operator who replaces the default keeps that responsibility, as with the UI.

What stays exposed on our side is argv — --conf pairs are readable in ps (SparkProcessBuilder.scala:133-157) — and any Spark line without the fix.

If we want the SparkConf copies gone as well, the mechanism is already in the tree. spark.kyuubi.* is copied into the engine's KyuubiConf at SparkSQLEngine.scala:315-318, and ZK auth reads the digest from KyuubiConf rather than SparkConf (ZookeeperClientProvider.scala:77), so blanking the SparkConf entry before getOrCreate — what :329-331 already does for engine credentials — is safe for the digest.

Does that change your answer to open question 2? With the RPC path closed upstream, the generated file buys only ps, and --conf with a per-engine token may be enough for the first slice.

[alexandrefimov]

The gateway import reached the routing core: apache/spark-connect-gateway#3 adds crates/routing/src/lib.rs, and the extension point that USER share level needs is in it. SelectionStrategy::select takes the full SessionKey, and BackendMember carries a free-form labels map, so a strategy can match the session identity against a backend's owner. The default RoundRobinStrategy ignores both — its doc comment says so — and the comment on BackendMember notes that the K8s watcher supplies addresses only, with pod-label surfacing still to do.

That settles the sub-question I parked in the design, whether the engine attributes need an explicit owner label. The two shipping pools are static config and a K8s Endpoints watch, and neither reads our ZK path, so the owner has to reach the gateway as a member label from whatever discovery source it runs on — BackendMember.labels is where a strategy would read it. I'd take the user-aware strategy and the label surfacing to that repo once the crate is merged.

On the Kyuubi slice, open question 2 I can answer myself now. Once SPARK-58658 lands, the Config RPC read-back is closed for our keys by the spark.redaction.regex append in SparkSQLEngine.scala:273-276, so a generated token file buys only ps hygiene; my default for the first slice is a per-engine token via --conf and no file. Questions 1, 3 and 4 are still yours to call.

Shall I start on the engine-side tasks and bring 1, 3 and 4 as concrete choices in the PR, or would you rather settle them here first?