Skip to content

[LIVY-1067] Bump CI Python to 3.11.11 and fix python-api PEP 440 version #540

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
roczei wants to merge 1 commit into
base: master
Choose a base branch
from
+22 −4
Open

[LIVY-1067] Bump CI Python to 3.11.11 and fix python-api PEP 440 version #540

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion .github/workflows/integration-tests.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,13 @@ jobs:
restore-keys: |
${{ runner.os }}-maven-
-
# Must match the pyenv-installed version in dev/docker/livy-dev-base/Dockerfile.
# Per each Spark release's python/setup.py: Spark 4.1 supports 3.10..3.14
# (python_requires=">=3.10"), Spark 3.5 supports 3.8..3.11
# (python_requires=">=3.8"). The intersection is 3.10 and 3.11; 3.11.11
# is picked as the shared version.
name: Set Python 3 as default
run: pyenv global 3 && echo "PYSPARK_PYTHON=$(which python3)" >> "$GITHUB_ENV"
run: pyenv global 3.11.11 && echo "PYSPARK_PYTHON=$(which python3)" >> "$GITHUB_ENV"
-
name: Set JDK version
run: update-alternatives --set java ${{ matrix.jdk_path }}
Expand Down
13 changes: 11 additions & 2 deletions dev/docker/livy-dev-base/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -70,8 +70,17 @@ RUN git clone https://github.com/pyenv/pyenv.git $HOME/pyenv
ENV PYENV_ROOT=$HOME/pyenv
ENV PATH="$HOME/pyenv/shims:$HOME/pyenv/bin:$HOME/bin:$PATH"

RUN pyenv install -v 3.9.21 && \
pyenv global 3.9.21 && \
# Python 3.11.11 is chosen because it is in the officially supported range of
# BOTH matrix profiles, per each Spark release's python/setup.py:
# * -Pspark4 (Spark 4.1.2): python_requires=">=3.10", classifiers list
# 3.10 / 3.11 / 3.12 / 3.13 / 3.14.
# * -Pspark3 (Spark 3.5.6): python_requires=">=3.8", classifiers list
# 3.8 / 3.9 / 3.10 / 3.11.
# The intersection is 3.10 and 3.11; we pick the higher one (3.11.11) so the
# same image serves both profiles. When bumping this, keep
# .github/workflows/integration-tests.yaml's `pyenv global` in sync.
RUN pyenv install -v 3.11.11 && \
pyenv global 3.11.11 && \
pyenv rehash

# Install build dependencies for python3
Expand Down
6 changes: 5 additions & 1 deletion python-api/setup.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,11 @@

setup(
name='livy-python-api',
version="1.0.0-SNAPSHOT",
# PEP 440 disallows Maven-style `-SNAPSHOT` suffixes; use `.dev0` instead,
# which is the canonical Python equivalent for "pre-release under active
# development". Modern pip (>= 24) rejects `1.0.0-SNAPSHOT` outright with
# `Invalid version: '1.0.0-SNAPSHOT'` when parsing this package.
version="1.0.0.dev0",
packages=["livy", "livy-tests"],
package_dir={
"": "src/main/python",
Expand Down
Loading