From 69440b615fb4efab38d361fe6920f349ef577480 Mon Sep 17 00:00:00 2001 From: Gabor Roczei <1918366+roczei@users.noreply.github.com> Date: Mon, 10 Aug 2026 16:49:34 +0200 Subject: [PATCH] [LIVY-1067] Bump CI Python to 3.11.11 and fix python-api PEP 440 version ## What changes were proposed in this pull request? Prepares the CI environment and the python-api package metadata for the upcoming Spark 4 support commit. Both changes are also useful on the existing Spark 3 build: the PEP 440 fix unblocks `pip3 install livy-python-api` on modern pip (>= 24), and the Python bump keeps the CI image on a version supported by both Spark 3.5 and Spark 4.1. **CI Python bumped to 3.11.11** dev/docker/livy-dev-base/Dockerfile pyenv Python bumped from 3.9.21 to 3.11.11, and .github/workflows/integration-tests.yaml pins `pyenv global 3.11.11` explicitly. Rationale (per each Spark release's python/setup.py): Spark 4.1 declares `python_requires=">=3.10"` with classifiers listing 3.10/3.11/3.12/3.13/3.14, while Spark 3.5 declares `python_requires=">=3.8"` with classifiers listing 3.8/3.9/3.10/3.11. The intersection of the two supported ranges is 3.10 and 3.11; we pick the higher one (3.11.11) so the same image serves both matrix profiles. The old 3.9.21 was below the Spark 4.1 lower bound. Both the Dockerfile and the workflow now document this choice inline so future bumps stay in sync. The CI Docker image (`ghcr.io/${owner}/livy-ci:latest`) had to be rebuilt and pushed manually for the PR CI to pick up the new Python before this PR is merged: `.github/workflows/build-ci-image.yaml` only fires on push to master, so branch/PR runs would otherwise still pull the stale image cached from the previous Dockerfile. Commands used (from macOS Apple Silicon, cross-built for linux/amd64 to match the GitHub Actions runners): gh auth refresh --scopes write:packages,read:packages gh auth token | docker login ghcr.io -u --password-stdin docker buildx build --platform linux/amd64 \ -t ghcr.io//livy-ci:latest --push \ dev/docker/livy-dev-base After this PR merges to master, the workflow will republish the image on any future Dockerfile change automatically, so this manual step is only needed for the bootstrap run. **python-api/setup.py PEP 440 version fix** Version bumped from `1.0.0-SNAPSHOT` (Maven-style, not PEP 440 compliant) to `1.0.0.dev0` (the canonical Python "pre-release under active development" form). Modern pip (>= 24) refuses to parse the old value with `Invalid version: '1.0.0-SNAPSHOT'`, which surfaced as `WARNING: Error parsing dependencies of livy-python-api` on every `pip3 install` and blocked the `validate Python-API requests` integration test from resolving its dependencies. The Maven POM version (`python-api/pom.xml`) stays `1.0.0-SNAPSHOT` -- Maven and pip have separate versioning conventions and only the pip-visible metadata needs to change. ## How was this patch tested? - `pip3 install ./python-api` no longer emits `WARNING: Error parsing dependencies of livy-python-api: Invalid version: '1.0.0-SNAPSHOT'`; the package installs cleanly on pip 24+. - Rebuilt the CI Docker image locally with the pyenv 3.11.11 bump and confirmed `python3 --version` reports `Python 3.11.11` inside the container. - GitHub Actions Integration Tests workflow runs `pyenv global 3.11.11` successfully on the -Pspark3 -Pscala-2.12 matrix entry (this commit does not yet add any Spark 4 entry). ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7) --- .github/workflows/integration-tests.yaml | 7 ++++++- dev/docker/livy-dev-base/Dockerfile | 13 +++++++++++-- python-api/setup.py | 6 +++++- 3 files changed, 22 insertions(+), 4 deletions(-) diff --git a/.github/workflows/integration-tests.yaml b/.github/workflows/integration-tests.yaml index 968937e0c..c1dd01b94 100644 --- a/.github/workflows/integration-tests.yaml +++ b/.github/workflows/integration-tests.yaml @@ -48,8 +48,13 @@ jobs: restore-keys: | ${{ runner.os }}-maven- - + # Must match the pyenv-installed version in dev/docker/livy-dev-base/Dockerfile. + # Per each Spark release's python/setup.py: Spark 4.1 supports 3.10..3.14 + # (python_requires=">=3.10"), Spark 3.5 supports 3.8..3.11 + # (python_requires=">=3.8"). The intersection is 3.10 and 3.11; 3.11.11 + # is picked as the shared version. name: Set Python 3 as default - run: pyenv global 3 && echo "PYSPARK_PYTHON=$(which python3)" >> "$GITHUB_ENV" + run: pyenv global 3.11.11 && echo "PYSPARK_PYTHON=$(which python3)" >> "$GITHUB_ENV" - name: Set JDK version run: update-alternatives --set java ${{ matrix.jdk_path }} diff --git a/dev/docker/livy-dev-base/Dockerfile b/dev/docker/livy-dev-base/Dockerfile index 5686e61d1..b0d06b735 100644 --- a/dev/docker/livy-dev-base/Dockerfile +++ b/dev/docker/livy-dev-base/Dockerfile @@ -70,8 +70,17 @@ RUN git clone https://github.com/pyenv/pyenv.git $HOME/pyenv ENV PYENV_ROOT=$HOME/pyenv ENV PATH="$HOME/pyenv/shims:$HOME/pyenv/bin:$HOME/bin:$PATH" -RUN pyenv install -v 3.9.21 && \ - pyenv global 3.9.21 && \ +# Python 3.11.11 is chosen because it is in the officially supported range of +# BOTH matrix profiles, per each Spark release's python/setup.py: +# * -Pspark4 (Spark 4.1.2): python_requires=">=3.10", classifiers list +# 3.10 / 3.11 / 3.12 / 3.13 / 3.14. +# * -Pspark3 (Spark 3.5.6): python_requires=">=3.8", classifiers list +# 3.8 / 3.9 / 3.10 / 3.11. +# The intersection is 3.10 and 3.11; we pick the higher one (3.11.11) so the +# same image serves both profiles. When bumping this, keep +# .github/workflows/integration-tests.yaml's `pyenv global` in sync. +RUN pyenv install -v 3.11.11 && \ + pyenv global 3.11.11 && \ pyenv rehash # Install build dependencies for python3 diff --git a/python-api/setup.py b/python-api/setup.py index 4d7b9b5f2..df1a4de1e 100644 --- a/python-api/setup.py +++ b/python-api/setup.py @@ -37,7 +37,11 @@ setup( name='livy-python-api', - version="1.0.0-SNAPSHOT", + # PEP 440 disallows Maven-style `-SNAPSHOT` suffixes; use `.dev0` instead, + # which is the canonical Python equivalent for "pre-release under active + # development". Modern pip (>= 24) rejects `1.0.0-SNAPSHOT` outright with + # `Invalid version: '1.0.0-SNAPSHOT'` when parsing this package. + version="1.0.0.dev0", packages=["livy", "livy-tests"], package_dir={ "": "src/main/python",