[KYUUBI #3356][SERVER] Add experimental Arrow Flight SQL frontend - #7624
Open
nryanov wants to merge 1 commit into
Open
[KYUUBI #3356][SERVER] Add experimental Arrow Flight SQL frontend#7624nryanov wants to merge 1 commit into
nryanov wants to merge 1 commit into
Conversation
### Why are the changes needed?
Add an experimental Arrow Flight SQL frontend so clients can run SQL and
metadata operations through Kyuubi while reusing the existing session/engine
lifecycle (Spark, Trino, Hive, JDBC, etc.). The producer is a transport
adapter only: it calls BackendService and does not invoke engine APIs
directly.
Main capabilities:
- Enable via `kyuubi.frontend.protocols=FLIGHT_SQL` and configure with
`kyuubi.frontend.flight.sql.*` keys (default bind port `10299`).
- Statement queries, catalogs/schemas/tables/table-types/type info, Arrow
IPC streaming with bounded page fetch, cancellation/cleanup.
- Auth: NONE, Basic/LDAP, Kerberos SPNEGO bootstrap with short-lived bearer
tokens; TLS via PEM (optional keystore→PEM materialization).
- HA: dedicated discovery namespace (`kyuubi.ha.flight.sql.namespace`),
advertised Flight endpoint, node-affine tickets (no transparent mid-query
failover).
- Flight-specific metrics.
Out of scope / intentional limits:
- Not end-to-end zero-copy past the existing BackendService Thrift/TRowSet
Arrow boundary.
- Explicit unsupported Flight SQL paths (ingest, Substrait, transactions,
full prepared-statement binding) return unimplemented status.
### How was this patch tested?
- Unit/integration tests under `kyuubi-server`:
- `KyuubiFlightSqlFrontendServiceSuite` (lifecycle / protocol enablement)
- `KyuubiFlightSqlQuerySuite` (execute + Arrow stream via Java
`FlightSqlClient`; run with Slow tag enabled)
- `KyuubiFlightAuthHandlerSuite`
- `KyuubiFlightArrowUtilsSuite`
- Shared harness: `WithFlightSqlServer`
- Config/HA coverage: `KyuubiConfSuite`, `HighAvailabilityConfSuite`
- Regenerated `docs/configuration/settings.md` via
`dev/gen/gen_all_config_docs.sh`
- Spotless formatting on touched modules
### Was this patch authored or co-authored using generative AI tooling?
Yes. Assisted-by: Cursor
Co-authored-by: Cursor <cursoragent@cursor.com>
github-actions
Bot
added
kind:documentation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Previous existing PR was closed, but feature-request is still active :)
Why are the changes needed?
Add an experimental Arrow Flight SQL frontend so clients can run SQL and metadata operations through Kyuubi while reusing the existing session/engine lifecycle (Spark, Trino, Hive, JDBC, etc.). The producer is a transport adapter only: it calls BackendService and does not invoke engine APIs directly.
Main capabilities:
kyuubi.frontend.protocols=FLIGHT_SQLand configure withkyuubi.frontend.flight.sql.*keys (default bind port10299).kyuubi.ha.flight.sql.namespace), advertised Flight endpoint, node-affine tickets (no transparent mid-query failover).Out of scope / intentional limits:
How was this patch tested?
kyuubi-server:KyuubiFlightSqlFrontendServiceSuite(lifecycle / protocol enablement)KyuubiFlightSqlQuerySuite(execute + Arrow stream via JavaFlightSqlClient; run with Slow tag enabled)KyuubiFlightAuthHandlerSuiteKyuubiFlightArrowUtilsSuiteWithFlightSqlServerKyuubiConfSuite,HighAvailabilityConfSuiteWas this patch authored or co-authored using generative AI tooling?
Yes. Assisted-by: Cursor Grok 4.5