Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/license.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:
- run: |
build/mvn org.apache.rat:apache-rat-plugin:check \
-Ptpcds -Pkubernetes-it \
-Pspark-3.3 -Pspark-3.4 -Pspark-3.5 -Pspark-4.0 -Pspark-4.1 -Pspark-4.2
-Pspark-3.5 -Pspark-4.0 -Pspark-4.1 -Pspark-4.2
- name: Upload rat report
if: failure()
uses: actions/upload-artifact@v7
Expand Down
25 changes: 0 additions & 25 deletions .github/workflows/master.yml
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,6 @@ jobs:
python:
- '3.9'
spark:
- '3.3'
- '3.4'
- '3.5'
spark-archive: [""]
exclude-tags: [""]
Expand All @@ -80,18 +78,6 @@ jobs:
spark-archive: '-Pscala-2.13'
exclude-tags: '-Dmaven.plugin.scalatest.exclude.tags=org.scalatest.tags.Slow,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.PaimonTest,org.apache.kyuubi.tags.HudiTest'
comment: 'normal'
- java: 8
python: '3.9'
spark: '3.5'
spark-archive: '-Dspark.archive.mirror=https://www.apache.org/dyn/closer.lua/spark/spark-3.3.3 -Dspark.archive.name=spark-3.3.3-bin-hadoop3.tgz -Pzookeeper-3.6'
exclude-tags: '-Dmaven.plugin.scalatest.exclude.tags=org.scalatest.tags.Slow,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.PaimonTest,org.apache.kyuubi.tags.HudiTest,org.apache.kyuubi.tags.SparkLocalClusterTest'
comment: 'verify-on-spark-3.3-binary'
- java: 8
python: '3.9'
spark: '3.5'
spark-archive: '-Dspark.archive.mirror=https://www.apache.org/dyn/closer.lua/spark/spark-3.4.4 -Dspark.archive.name=spark-3.4.4-bin-hadoop3.tgz -Pzookeeper-3.6'
exclude-tags: '-Dmaven.plugin.scalatest.exclude.tags=org.scalatest.tags.Slow,org.apache.kyuubi.tags.DeltaTest,org.apache.kyuubi.tags.IcebergTest,org.apache.kyuubi.tags.PaimonTest,org.apache.kyuubi.tags.SparkLocalClusterTest'
comment: 'verify-on-spark-3.4-binary'
- java: 17
python: '3.11'
spark: '3.5'
Expand Down Expand Up @@ -248,17 +234,6 @@ jobs:
fail-fast: false
max-parallel: 1
matrix:
java:
- 17
scala:
- '2.12'
- '2.13'
spark-compile:
- '3.5'
spark-runtime:
- '3.4'
- '3.3'
comment: [ "normal" ]
include:
- java: 17
scala: "2.13"
Expand Down
18 changes: 7 additions & 11 deletions .github/workflows/publish-snapshot-nexus.yml
Original file line number Diff line number Diff line change
Expand Up @@ -31,27 +31,23 @@ jobs:
matrix:
branch:
- master
- branch-1.8
- branch-1.9
- branch-1.12
profiles:
- -Pflink-provided,spark-provided,hive-provided,spark-3.3
- -Pflink-provided,spark-provided,hive-provided,spark-3.4,tpcds
include:
- branch: master
profiles: -Pflink-provided,spark-provided,hive-provided,spark-3.5
- branch: branch-1.9
profiles: -Pflink-provided,spark-provided,hive-provided,spark-3.5
- -Pflink-provided,spark-provided,hive-provided,spark-3.5,tpcds
- -Pflink-provided,spark-provided,hive-provided,scala-2.13,spark-4.0
- -Pflink-provided,spark-provided,hive-provided,scala-2.13,spark-4.1
- -Pflink-provided,spark-provided,hive-provided,scala-2.13,spark-4.2,tpcds
steps:
- uses: actions/checkout@v7
with:
ref: ${{ matrix.branch }}
- name: Free up disk space
run: ./.github/scripts/free_disk_space.sh
- name: Setup JDK 8
- name: Setup JDK 17
uses: actions/setup-java@v5
with:
distribution: temurin
java-version: 8
java-version: 17
cache: 'maven'
- name: Publish Snapshot Jar to Nexus - ${{ matrix.branch }}
env:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/python.yml
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,7 @@ jobs:
fail-fast: false
max-parallel: 15
matrix:
python-version: ["3.8", "3.9", "3.10"]
python-version: ["3.9", "3.10"]
env:
PYTHONHASHSEED: random
steps:
Expand Down
4 changes: 1 addition & 3 deletions .github/workflows/style.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ jobs:
max-parallel: 15
matrix:
profiles:
- '-Pflink-provided,hive-provided,spark-provided,spark-3.5,spark-3.4,spark-3.3,tpcds,kubernetes-it'
- '-Pflink-provided,hive-provided,spark-provided,spark-3.5,tpcds,kubernetes-it'

steps:
- uses: actions/checkout@v7
Expand Down Expand Up @@ -66,8 +66,6 @@ jobs:
run: |
export MAVEN_ARGS="-DskipTests -Dorg.slf4j.simpleLogger.defaultLogLevel=warn -Drat.skip=true -Dmaven.scalastyle.skip=true -Dspotless.check.skip=true -Pflink-provided,hive-provided,spark-provided"
build/mvn clean install -Ptpcds
build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-3-3 -am -Pspark-3.3
build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-3-4 -am -Pspark-3.4
build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-3-5,extensions/spark/kyuubi-spark-connector-hive -am -Pspark-3.5
build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-4-0 -am -Pspark-4.0 -Pscala-2.13
build/mvn clean install -pl extensions/spark/kyuubi-extension-spark-4-1 -am -Pspark-4.1 -Pscala-2.13
Expand Down
16 changes: 8 additions & 8 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,14 +67,14 @@ Use `-am` (also-make) when building or testing a single module — without it, M

### Engine profile matrix

| Profile | Notes |
|---|---|
| `-Pspark-3.5` (default), `-Pspark-{3.3,3.4,4.0,4.1,master}` | Spark version |
| `-Pflink-1.20` (default), `-Pflink-{1.17,1.18,1.19}` | Flink version |
| `-Pscala-2.13` | Scala 2.13 (default is 2.12) |
| `-P{spark,flink,hive}-provided` | skip bundled engine downloads |
| `-Pmirror-cdn` | use Apache mirror CDN for engine archives |
| `-Pfast` | skip tests/style/docs/enforcer/RAT/downloads |
| Profile | Notes |
|---------------------------------------------------------|----------------------------------------------|
| `-Pspark-3.5` (default), `-Pspark-{4.0,4.1,4.2,master}` | Spark version |
| `-Pflink-1.20` (default), `-Pflink-{1.17,1.18,1.19}` | Flink version |
| `-Pscala-2.13` | Scala 2.13 (default is 2.12) |
| `-P{spark,flink,hive}-provided` | skip bundled engine downloads |
| `-Pmirror-cdn` | use Apache mirror CDN for engine archives |
| `-Pfast` | skip tests/style/docs/enforcer/RAT/downloads |

When engine code varies across versions, gate source/binary differences by Maven profile and runtime capability differences by feature detection — not by parsing version strings.

Expand Down
10 changes: 0 additions & 10 deletions build/release/release.sh
Original file line number Diff line number Diff line change
Expand Up @@ -110,16 +110,6 @@ upload_svn_staging() {
}

upload_nexus_staging() {
# Spark Extension Plugin for Spark 3.3 and Scala 2.12
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided,hive-provided,spark-3.3 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl extensions/spark/kyuubi-extension-spark-3-3 -am

# Spark Extension Plugin for Spark 3.4 and Scala 2.12
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided,hive-provided,spark-3.4 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
-pl extensions/spark/kyuubi-extension-spark-3-4 -am

# Spark Extension Plugin for Spark 4.0 and Scala 2.13
${KYUUBI_DIR}/build/mvn clean deploy -DskipTests -Papache-release,flink-provided,spark-provided,hive-provided,spark-4.0,scala-2.13 \
-s "${KYUUBI_DIR}/build/release/asf-settings.xml" \
Expand Down
50 changes: 0 additions & 50 deletions dev/kyuubi-codecov/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -155,56 +155,6 @@
</build>

<profiles>
<profile>
<id>spark-3.3</id>
<dependencies>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-extension-spark-3-3_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-connector-hive_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-authz_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-lineage_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</profile>
<profile>
<id>spark-3.4</id>
<dependencies>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-extension-spark-3-4_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-connector-hive_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-authz_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kyuubi</groupId>
<artifactId>kyuubi-spark-lineage_${scala.binary.version}</artifactId>
<version>${project.version}</version>
</dependency>
</dependencies>
</profile>
<profile>
<id>spark-3.5</id>
<dependencies>
Expand Down
2 changes: 1 addition & 1 deletion dev/reformat
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ set -x

KYUUBI_HOME="$(cd "`dirname "$0"`/.."; pwd)"

PROFILES="-Pflink-provided,hive-provided,spark-provided,spark-4.2,spark-4.1,spark-4.0,spark-3.5,spark-3.4,spark-3.3,tpcds,kubernetes-it"
PROFILES="-Pflink-provided,hive-provided,spark-provided,spark-4.2,spark-4.1,spark-4.0,spark-3.5,tpcds,kubernetes-it"

# python style checks rely on `black` in path
if ! command -v black &> /dev/null
Expand Down
18 changes: 12 additions & 6 deletions docs/contributing/code/building.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,19 +61,25 @@ mvn clean install -pl '!dev/kyuubi-codecov,!kyuubi-assembly' -DskipTests

Since v1.1.0, Kyuubi support building with different Spark profiles,

| Profile | Default | Since |
|-------------|---------|-------|
| -Pspark-3.3 | | 1.6.0 |
| -Pspark-3.4 | | 1.8.0 |
| -Pspark-3.5 | ✓ | 1.8.0 |
| Profile | Default | Since | Removed |
|-------------|---------|--------|---------|
| -Pspark-3.0 | | 1.1.0 | 1.6.0 |
| -Pspark-3.1 | | 1.1.0 | 1.9.0 |
| -Pspark-3.2 | | 1.4.0 | 1.10.0 |
| -Pspark-3.3 | | 1.6.0 | 1.13.0 |
| -Pspark-3.4 | | 1.8.0 | 1.13.0 |
| -Pspark-3.5 | ✓ | 1.8.0 | |
| -Pspark-4.0 | | 1.10.0 | |
| -Pspark-4.1 | | 1.11.0 | |
| -Pspark-4.2 | | 1.12.0 | |

## Building Kyuubi Against Different Scala Versions

Since v1.8.0, Kyuubi support building with different Scala profile. Currently, Kyuubi supports building with Scala 2.12 and 2.13, while Scala 2.12 by default.

| Profile | Default | Since |
|--------------|---------|-------|
| (Scala 2.12) | ✓ | - |
| -Pscala-2.12 | ✓ | - |
| -Pscala-2.13 | | 1.8.0 |

Please activate `scala-2.13` profile when Scala 2.13 support is needed. The GA tests have covered integration test with the Kyuubi server, engines and related plugins, while the Flink engine and it's integration tests are not included for the reason that Flink does not support Scala 2.13 yet and will pull out client support for Scala.
Expand Down
4 changes: 4 additions & 0 deletions docs/deployment/migration-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,10 @@

# Kyuubi Migration Guide

## Upgrading from Kyuubi 1.12 to 1.13

* Since Kyuubi 1.13, the support of Spark engine for Spark 3.3 and 3.4 is removed.

## Upgrading from Kyuubi 1.11 to 1.12

* Since Kyuubi 1.12, the support of variable `<KYUUBI_HOME>` substitution in config `kyuubi.metadata.store.jdbc.url` is deprecated, use `{{KYUUBI_HOME}}` instead.
Expand Down
2 changes: 1 addition & 1 deletion docs/extensions/engines/spark/jvm-quake.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ So introducing JVMQuake provides more granular monitoring of GC behavior, enabli

## Usage

JVM Quake is implemented through Spark plugins, This plugin technically supports Spark 3.0 onwards, but was only verified with Spark 3.3 to 4.0 in CI.
JVM Quake is implemented through Spark plugins, This plugin technically supports Spark 3.0 onwards, but was only verified with Spark 3.5 to 4.2 in CI.

### Build with Apache Maven

Expand Down
24 changes: 12 additions & 12 deletions docs/extensions/engines/spark/lineage.md
Original file line number Diff line number Diff line change
Expand Up @@ -122,18 +122,18 @@ build/mvn clean package -pl :kyuubi-spark-lineage_2.12 -am -DskipTests -Dspark.v

The available `spark.version`s are shown in the following table.

| Spark Version | Supported | Remark |
|:-------------:|:---------:|:------:|
| master | √ | - |
| 3.5.x | √ | - |
| 3.4.x | √ | - |
| 3.3.x | √ | - |
| 3.2.x | √ | - |
| 3.1.x | x | - |
| 3.0.x | x | - |
| 2.4.x | x | - |

Currently, Spark released with Scala 2.12 are supported.
| Spark Version | Supported |
|:-------------:|:---------:|
| 4.2.x | √ |
| 4.1.x | √ |
| 4.0.x | √ |
| 3.5.x | √ |
| 3.4.x | x |
| 3.3.x | x |
| 3.2.x | x |
| 3.1.x | x |
| 3.0.x | x |
| 2.4.x | x |

### Test with ScalaTest Maven plugin

Expand Down
24 changes: 12 additions & 12 deletions docs/extensions/engines/spark/rules.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@

# Auxiliary Optimization Rules

Kyuubi provides SQL extension out of box. Due to the version compatibility with Apache Spark, currently we support Apache Spark branch-3.1 and later.
Kyuubi provides SQL extension out of box. Due to the version compatibility with Apache Spark, currently we support Apache Spark 3.5 and later.
And don't worry, Kyuubi will support the new Apache Spark version in the future. Thanks to the adaptive query execution framework (AQE), Kyuubi can do these optimizations.

## Features
Expand Down Expand Up @@ -57,20 +57,20 @@ And don't worry, Kyuubi will support the new Apache Spark version in the future.

## Usage

| Kyuubi Spark SQL extension | Supported Spark version(s) | Available since | EOL | Bundled in Binary release tarball | Maven profile |
|----------------------------|----------------------------|------------------|-------|-----------------------------------|---------------|
| kyuubi-extension-spark-3-1 | 3.1.x | 1.3.0-incubating | 1.8.0 | 1.3.0-incubating | spark-3.1 |
| kyuubi-extension-spark-3-2 | 3.2.x | 1.4.0-incubating | 1.9.0 | 1.4.0-incubating | spark-3.2 |
| kyuubi-extension-spark-3-3 | 3.3.x | 1.6.0-incubating | N/A | 1.6.0-incubating | spark-3.3 |
| kyuubi-extension-spark-3-4 | 3.4.x | 1.8.0 | N/A | 1.8.0 | spark-3.4 |
| kyuubi-extension-spark-3-5 | 3.5.x | 1.8.0 | N/A | 1.9.0 | spark-3.5 |
| kyuubi-extension-spark-4-0 | 4.0.x | 1.10.0 | N/A | 1.10.0 | spark-4.0 |
| kyuubi-extension-spark-4-1 | 4.1.x | 1.11.0 | N/A | 1.11.0 | spark-4.1 |
| kyuubi-extension-spark-4-2 | 4.2.x | 1.12.0 | N/A | 1.12.0 | spark-4.2 |
| Kyuubi Spark SQL extension | Supported Spark version(s) | Available since | EOL | Bundled in Binary release tarball | Maven profile |
|----------------------------|----------------------------|------------------|--------|-----------------------------------|---------------|
| kyuubi-extension-spark-3-1 | 3.1.x | 1.3.0-incubating | 1.8.0 | 1.3.0-incubating - 1.4.1 | spark-3.1 |
| kyuubi-extension-spark-3-2 | 3.2.x | 1.4.0-incubating | 1.9.0 | 1.5.0 - 1.6.1 | spark-3.2 |
| kyuubi-extension-spark-3-3 | 3.3.x | 1.6.0-incubating | 1.12.0 | N/A | spark-3.3 |
| kyuubi-extension-spark-3-4 | 3.4.x | 1.8.0 | 1.12.0 | N/A | spark-3.4 |
| kyuubi-extension-spark-3-5 | 3.5.x | 1.8.0 | N/A | N/A | spark-3.5 |
| kyuubi-extension-spark-4-0 | 4.0.x | 1.10.0 | N/A | N/A | spark-4.0 |
| kyuubi-extension-spark-4-1 | 4.1.x | 1.11.0 | N/A | N/A | spark-4.1 |
| kyuubi-extension-spark-4-2 | 4.2.x | 1.12.0 | N/A | N/A | spark-4.2 |

1. Check the matrix that if you are using the supported Spark version, and find the corresponding Kyuubi Spark SQL Extension jar
2. Get the Kyuubi Spark SQL Extension jar
1. Each Kyuubi binary release tarball only contains one default version of Kyuubi Spark SQL Extension jar, if you are looking for such version, you can find it under `$KYUUBI_HOME/extension`
1. Kyuubi binary release tarballs bundled one default version of the Kyuubi Spark SQL Extension jar until 1.6.1, and it is no longer bundled since 1.7.0
2. All supported versions of Kyuubi Spark SQL Extension jar will be deployed to [Maven Central](https://search.maven.org/search?q=kyuubi-extension-spark)
3. If you like, you can compile Kyuubi Spark SQL Extension jar by yourself, please activate the corresponding Maven's profile on you compile command, i.e. you can get Kyuubi Spark SQL Extension jar for Spark 3.5 under `extensions/spark/kyuubi-extension-spark-3-5/target` when compile with `-Pspark-3.5`
3. Put the Kyuubi Spark SQL extension jar `kyuubi-extension-spark-*.jar` into `$SPARK_HOME/jars`
Expand Down
4 changes: 2 additions & 2 deletions docs/quick_start/quick_start.rst
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ pre-installed and the ``JAVA_HOME`` is correctly set to each component.
**Kyuubi** Gateway \ |release| \ - Kyuubi Server
Engine lib - Kyuubi Engine
Beeline - Kyuubi Beeline
**Spark** Engine 3.3 to 3.5, 4.0 to 4.2 A Spark distribution
**Spark** Engine 3.5, 4.0 to 4.2 A Spark distribution
**Flink** Engine 1.17 to 1.20, 2.0 to 2.3 A Flink distribution
**Trino** Engine N/A A Trino cluster allows to access via trino-client v411
**Doris** Engine N/A A Doris cluster
Expand Down Expand Up @@ -146,7 +146,7 @@ To install Spark, you need to unpack the tarball. For example,

.. code-block::

$ tar zxf spark-3.4.2-bin-hadoop3.tgz
$ tar zxf spark-3.5.8-bin-hadoop3.tgz

Configuration
~~~~~~~~~~~~~
Expand Down
Loading
Loading