[FEATURE] 能否考虑增加一个网关之后路由之前的一个钩子

[PromiseOne]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the feature

我想要在kyuubi网关中，添加一个用于在SQL路由到不同的执行引擎之前的一个钩子，其目的是可以由用户自行对其提交的SQL进行处理（例如鉴权，风险SQL拦截）等。我已知道目前可以对sparksql支持authz的一个鉴权，但是既然作为一个统一的大数据SQL网关，完全可以将这部分能力下方给用户，有了这个钩子，用户可以针对提交的sql做更多想做的事情。另外，是否还可以提供一个用于处理结果集的钩子，其目的是为了在返回的结果集中，可以对其结果集中的敏感数据进行加密，要知道现在很多企业中，仍然有很多明文信息，有了这个钩子，用户可以轻易的对其结果集进行二次处理。

Motivation

No response

Describe the solution

No response

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
• No. I cannot submit a PR at this time.

[github-actions]

Hello @PromiseOne,
Thanks for finding the time to report the issue!
We really appreciate the community's efforts to improve Apache Kyuubi.

[PromiseOne]

这个功能大概会在哪个版本推出呢？

[wangzhigang1999]

This is a reasonable and useful request 👍

Pushing pre-routing SQL processing down to users fills a real gap beyond the existing Spark Authz integration. As a unified SQL gateway, providing a single stable extension point at the gateway layer is a better fit than having each engine reinvent it — authorization, risky-SQL interception, and SQL rewriting can all be served by the same hook.

I plan to implement this in two parts:

1. Pre-routing statement hook — a server-side StatementInterceptor SPI invoked for each statement before operation creation and engine routing. It supports inspect / rewrite / reject, which covers both authorization and risky-SQL interception. Both the interactive and batch paths will go through it.
2. Result-set hook — post-processing on the returned result set, so sensitive columns can be masked or encrypted before they reach the client.

I'll take this one and open a PR. It will target the next feature release; the exact version will follow the milestone it lands in.

[PromiseOne]

这是一个合理且有用的请求 👍

将预路由SQL处理推送给用户，填补了现有Spark Authz集成之外的真正空白。作为统一的SQL网关，在网关层提供单一稳定的扩展点比每个引擎重新发明它更合适——授权、风险极高的SQL拦截和SQL重写都可以通过同一个钩子完成。

我计划分两部分实施：

1. 预路由语句钩子——在创建操作和引擎路由前，为每个语句调用的服务器端SPI。它支持检查/重写/拒绝，涵盖授权和风险 SQL 拦截。交互路径和批处理路径都会经过它。StatementInterceptor
2. 结果集钩子——对返回的结果集进行后处理，以便在敏感列到达客户端之前被掩蔽或加密。

我会接受这个并开永久居民。它将针对下一个功能发布;具体版本将根据其达到的里程碑而定。

👍，太赞了，有了这份扩展之后，ranger的替换变成了可能，同时也是在kyuubi的鉴权和数据安全体系补齐了至关重要的一块拼图。还望大佬到时候考虑到java语言的扩展，可以使java开发者可以更便捷的扩展功能。

[wangzhigang1999]

Where this stands

Quick update. The statement hook is working, and I prototyped the result hook too — but benchmarking it shifted my thinking on the scope, so let me lay out where I've landed.

First, on batch: I floated earlier that batch could go through the same hook, but it doesn't really hold up. A batch is just an app submission — the gateway only sees the submission parameters (resource, class, args, conf), not the SQL the application actually runs on the engine, and never the result rows. So the statement/result hooks only make sense on the interactive path. The most batch could ever have is a submission-level hook over the BatchRequest itself, which feels like a separate thing — happy to revisit that on its own later.

On the result hook: it's a general per-row transform (encryption, redaction, masking are just instances of it), sitting at KyuubiOperation.getNextRowSetInternal. The interesting part turned out to be the cost, which depends a lot on the result format. In a local microbenchmark (single-threaded, per fetch batch, Thrift columnar vs Arrow IPC, a wide row of 15 columns transforming 2 string columns) I saw:

rows/batch	Thrift	Arrow	slower
10K	~0.3 ms / ~2 MB	3.56 ms / 17 MB	11.9× / ~8× heap
100K	3.55 ms / 19.5 MB	47.2 ms / 154 MB	13.3× / ~8× heap

Numbers are from my own setup so treat them as indicative, but the shape is what the code predicts: with Thrift the server already holds the deserialized TRowSet, so a transform only touches the columns you actually change — sub-millisecond, basically noise. Arrow is forwarded opaquely (the engine packs the whole batch into a single binary column the server doesn't decode), so to touch even one column you have to decode and re-encode the entire batch, and that cost scales with the total column count rather than what you transform. On a wide table that's 12–20× slower and ~8× the heap churn per batch (the ratio peaks ~20× at 1K rows; a narrow 1-column row is only 2.6×). Under concurrency that's not something I'd want to hide silently behind a hook.

So for v1 I'd keep it tight: interactive path only, result transform on Thrift columnar, and fail-closed when a transform is configured and the operation returns Arrow — instead of quietly eating that cost. This isn't a global ban on Arrow; it only refuses the combination of an enabled transform plus an Arrow result. Arrow support can come later as an explicit opt-in, type-preserving only. The statement hook is ready, so I'll send that PR first and let the result transform follow.

The one thing I'd like a sanity check on before I commit to it is that fail-closed call — refusing only when a transform meets an Arrow result. Does that feel right to everyone?

[PromiseOne]

感谢你的详细设计和充分的 benchmark 数据，对性能影响的量化分析非常有说服力。我完全赞同在 v1 阶段采用 fail-closed 策略，这确实是避免用户“静默性能劣化”最稳妥的做法。
不过，顺着你的思路，我想到一个后续可能遇到的现实场景：假如用户已经基于安全合规要求，全局开启了 Result Transform Hook（比如强制数据脱敏），但业务上又确实存在一个需要极致性能的 Arrow 格式查询场景。
在这种情况下，如果采用硬性的 fail-closed，用户就陷入了“二选一”的困境：要么为了 Hook 放弃 Arrow 的性能，要么为了 Arrow 冒着违规风险取消 Hook。
所以我在想，是否可以引入一个细粒度的控制参数，可以控制 arrow 格式的使用。但是这个参数仅管理员配置（如果是session级别的话，会有被用户绕过的可能，我记得没错的话，应该管理员也可以控制哪些参数能够在session中使用）。
我考虑的场景是这样的：
场景1：交互式查询，有使用后置hook的可能（因为在金融行业，有数据安全等合规性要求），这种场景必然是使用 Thrift。
场景2：以kyuubi服务，提供一些高性能接口，那么这种场景就有可能使用到 arrow ，但是使用这种场景通常不是人使用，而是接口调用，这种场景下，可能性能就会成为优先考虑的问题。
所以，我觉得应该有一个参数对这个进行控制，虽然一个参数不能控制两种场景，但是管理员可以搭建多个kyuubi来应对不同的场景。
我个人想法还是想着尽可能考虑多种场景，以应对不同人的需要。