[KYUUBI #6995] Support Flink 2.0, 2.1, 2.2 and 2.3 - #7628
Closed
pan3793 wants to merge 5 commits into
Closed
Conversation
github-actions
Bot
added
kind:documentation
Assisted-by: Claude Opus 5
pan3793
force-pushed
the
kyuubi-flink-2
branch
from
August 12, 2026 15:39
e506b36 to
a0056ce
Compare
FLINK-14068 (2.0.0) removed Configuration#setString/setInteger, use set
FLINK-38974 (2.3.0) rejects jobs that carry no application id in the dispatcher
assume in beforeAll aborts the whole suite and is reported as failure, guard jdbcUrl instead so each test is canceled
pan3793
requested review from
link3280 and
wForget
and removed request for
link3280
August 13, 2026 05:14
Member
Author
|
After this, I'm going to drop support for Flink 1.17, 1.18, and 1.19, then mark Flink 2.0 as deprecated because it has been EOL |
pan3793
changed the title
wForget
approved these changes
Aug 13, 2026
FLINK-38974 (2.3.0) makes the dispatcher reject jobs that carry no application registered in it. Kyuubi statements run through the SQL gateway, which builds plain StreamExecutionEnvironments, so stamp the application id captured from the bootstrap job on every submitted StreamGraph. Assisted-by: Codex
Member
Author
|
Pushed a new commit to resolve the application mode with Flink 2.3 |
Member
Author
|
thanks, merged to master |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Closes #6995.
Extend the Flink SQL engine to support Flink 2.0, 2.1, 2.2 and 2.3, while keeping the existing
1.17 to 1.20 support. The engine keeps shipping a single jar that runs against all supported
Flink versions, so the version differences are bridged by reflection rather than by per-version
source trees.
Flink 2.x changes that the engine has to adapt to:
org.apache.flink.api.common.time.Timein favor ofDuration,which is a parameter of
EmbeddedJobClient's constructor.Configurationparameter toEmbeddedExecutor's constructor.EmbeddedExecutorFactory's andEmbeddedExecutor's job ids into theapplication, suspended and terminal ones, and made the dispatcher reject jobs that carry no
registered application id. Flink resolves the factory constructor at compile time, so Kyuubi's
copy of
EmbeddedExecutorFactorydeclares both constructors and stamps the application idcaptured from the bootstrap job on every submitted
StreamGraph, which enables applicationmode on Flink 2.3.
flink run-applicationintoflink runand removedthe former, so
FlinkProcessBuilderpicks the action by the version detected from theflink-distjar under$FLINK_HOME/lib.CliOptionsParser#checkFilePath, which is now inlined intoFlinkEngineUtils. Note thatDefaultContext#loadneeds no adaption, its dependency listchanged from
List<URL>toList<URI>but both erase to the same descriptor, and Flink stillreads the elements as
URLinSessionContext#create.Configuration#getString(ConfigOption)was removed,Configuration#get(ConfigOption)is usedinstead, which exists in all supported versions.
Flink 2.x requires Java 11 or higher and is compiled with Java 11 target, so the
flink-2.xprofiles raise
enforcer.maxJdkVersionto 11, and the Flink 2.x CI jobs run on Java 17.How was this patch tested?
New UTs cover the Flink version detection and the application mode command selection in
FlinkProcessBuilderSuite.FlinkOperationSuiteasserts the result ofENCODEby version, since FLINK-38062 (2.2.0)corrected its return type from
BINARYtoVARBINARY, before that the result was silentlytruncated to the first byte.
externals/kyuubi-flink-sql-enginetests were run against each Flink binary distribution, withFLINK_HOMEpinned to the matching distribution:FlinkOperationOnYarnSuitecovers application mode on Flink 2.3, and the cross-version jobcompiles the engine against Flink 1.20 and runs it on the Flink 2.3 distribution.
The GA
flink-itmatrix now always compiles against the default Flink version and varies onlythe runtime distribution, covering 1.17, 1.18, 1.19 on Java 8 and 2.0, 2.1, 2.2, 2.3 on Java 17.
Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude Opus 5, Codex