Code of Conduct
Search before asking
Describe the feature
Kyuubi already provides several useful Spark SQL auxiliary optimization rules, such as repartition/rebalance before writing, forcing shuffle before join for AQE skew optimization, final-stage config isolation, final write stage resource isolation, and SQL classification driven dynamic behavior.
These capabilities show that Kyuubi Spark extensions can already apply execution-side optimizations through Spark parser / analyzer / optimizer / planner / query-stage rules.
I would like to propose a more general capability on top of these existing extensions:
Support a configuration-driven query optimization runtime for Kyuubi Spark SQL extensions.
The idea is that Kyuubi should be able to consume a query-level optimization plan passed from upstream systems through configuration, and apply that plan using existing Spark SQL extension points.
This is not about moving optimization decision logic into Kyuubi.
Instead, it is about making Kyuubi a better execution-side runtime for upstream optimization systems.
A query-level optimization plan may include things such as:
write optimization strategy
repartition / coalesce guidance
AQE-related overrides
join strategy bias
resource isolation directives
trace / dual-run identifiers
This becomes increasingly useful in the AI-assisted SQL optimization era, where optimization suggestions may be generated by upstream rule engines, historical optimizers, or AI systems, but still need a standard and observable execution-side runtime to apply them safely.
Motivation
In many real-world deployments, query optimization decisions are often made by an upstream gateway or query platform, for example based on:
workload classification
historical workload analysis
workload-specific resource sizing
heavy query diagnosis
AI-assisted SQL optimization
dual-run validation before rollout
In such an architecture, the upstream system should remain the decision maker, while Kyuubi acts as the execution-side runtime.
Today, Kyuubi has multiple independent optimization rules, but there is no unified runtime abstraction for consuming a query-level optimization plan from upstream systems.
As a result:
upstream systems can only influence execution through scattered Spark confs
there is no common way to express optimization intent across different optimization rules
there is no unified execution trace showing:what optimization plan was received
which directives were applied
where they were applied
whether they were fully or partially applied
This is especially important for AI-assisted SQL optimization workflows, because AI-generated optimization suggestions need:
a standard runtime contract
execution-side observability
safe rollout support
dual-run / validation traceability
So the motivation is not only feature completeness, but also making Kyuubi more suitable as a general execution runtime for upstream optimization systems, including AI-assisted ones.
Describe the solution
A possible implementation could introduce a lightweight internal optimization runtime abstraction, for example:
- Unified optimization plan abstraction
An internal OptimizationPlan could represent things such as:
optimization enabled / disabled
write optimization strategy
repartition / coalesce guidance
AQE overrides
join strategy bias
resource isolation directives
trace id / source / dual-run id
This plan should be resolved from a well-defined set of Spark/Kyuubi configuration keys.
- Optimization plan resolver
A common resolver could parse raw conf into a structured OptimizationPlan, so that each rule does not parse raw string confs independently.
- Layered runtime application
Different optimization categories could be applied at different layers:
Session / query conf only
AQE toggles
shuffle partitions
advisory partition size
skew join switches
Post-hoc resolution / logical rewrite
write-side repartition / rebalance
final-stage tagging
logical hint-like guidance
Planner / query stage prep
resource profile injection
final-stage resource isolation
physical strategy bias
- Execution trace
Kyuubi could expose execution-side trace information such as:
what optimization plan was received
which directives were consumed
which rule/layer applied them
whether they were fully or partially applied
I think this should be built on top of existing Spark SQL extension points already used by Kyuubi, rather than introducing a completely separate mechanism.
Additional context
No response
Are you willing to submit PR?
Code of Conduct
Search before asking
Describe the feature
Kyuubi already provides several useful Spark SQL auxiliary optimization rules, such as repartition/rebalance before writing, forcing shuffle before join for AQE skew optimization, final-stage config isolation, final write stage resource isolation, and SQL classification driven dynamic behavior.
These capabilities show that Kyuubi Spark extensions can already apply execution-side optimizations through Spark parser / analyzer / optimizer / planner / query-stage rules.
I would like to propose a more general capability on top of these existing extensions:
Support a configuration-driven query optimization runtime for Kyuubi Spark SQL extensions.
The idea is that Kyuubi should be able to consume a query-level optimization plan passed from upstream systems through configuration, and apply that plan using existing Spark SQL extension points.
This is not about moving optimization decision logic into Kyuubi.
Instead, it is about making Kyuubi a better execution-side runtime for upstream optimization systems.
A query-level optimization plan may include things such as:
write optimization strategy
repartition / coalesce guidance
AQE-related overrides
join strategy bias
resource isolation directives
trace / dual-run identifiers
This becomes increasingly useful in the AI-assisted SQL optimization era, where optimization suggestions may be generated by upstream rule engines, historical optimizers, or AI systems, but still need a standard and observable execution-side runtime to apply them safely.
Motivation
In many real-world deployments, query optimization decisions are often made by an upstream gateway or query platform, for example based on:
workload classification
historical workload analysis
workload-specific resource sizing
heavy query diagnosis
AI-assisted SQL optimization
dual-run validation before rollout
In such an architecture, the upstream system should remain the decision maker, while Kyuubi acts as the execution-side runtime.
Today, Kyuubi has multiple independent optimization rules, but there is no unified runtime abstraction for consuming a query-level optimization plan from upstream systems.
As a result:
upstream systems can only influence execution through scattered Spark confs
there is no common way to express optimization intent across different optimization rules
there is no unified execution trace showing:what optimization plan was received
which directives were applied
where they were applied
whether they were fully or partially applied
This is especially important for AI-assisted SQL optimization workflows, because AI-generated optimization suggestions need:
a standard runtime contract
execution-side observability
safe rollout support
dual-run / validation traceability
So the motivation is not only feature completeness, but also making Kyuubi more suitable as a general execution runtime for upstream optimization systems, including AI-assisted ones.
Describe the solution
A possible implementation could introduce a lightweight internal optimization runtime abstraction, for example:
An internal OptimizationPlan could represent things such as:
optimization enabled / disabled
write optimization strategy
repartition / coalesce guidance
AQE overrides
join strategy bias
resource isolation directives
trace id / source / dual-run id
This plan should be resolved from a well-defined set of Spark/Kyuubi configuration keys.
A common resolver could parse raw conf into a structured OptimizationPlan, so that each rule does not parse raw string confs independently.
Different optimization categories could be applied at different layers:
Session / query conf only
AQE toggles
shuffle partitions
advisory partition size
skew join switches
Post-hoc resolution / logical rewrite
write-side repartition / rebalance
final-stage tagging
logical hint-like guidance
Planner / query stage prep
resource profile injection
final-stage resource isolation
physical strategy bias
Kyuubi could expose execution-side trace information such as:
what optimization plan was received
which directives were consumed
which rule/layer applied them
whether they were fully or partially applied
I think this should be built on top of existing Spark SQL extension points already used by Kyuubi, rather than introducing a completely separate mechanism.
Additional context
No response
Are you willing to submit PR?