[LIVY-1067] Bump CI Python to 3.11.11 and fix python-api PEP 440 version

[roczei]

What changes were proposed in this pull request?

Prepares the CI environment and the python-api package metadata for the upcoming Spark 4 support commit. Both changes are also useful on the existing Spark 3 build: the PEP 440 fix unblocks pip3 install livy-python-api on modern pip (>= 24), and the Python bump keeps the CI image on a version supported by both Spark 3.5 and Spark 4.1.

CI Python bumped to 3.11.11
dev/docker/livy-dev-base/Dockerfile pyenv Python bumped from 3.9.21 to 3.11.11, and .github/workflows/integration-tests.yaml pins pyenv global 3.11.11 explicitly. Rationale (per each Spark release's python/setup.py): Spark 4.1 declares python_requires=">=3.10" with classifiers listing 3.10/3.11/3.12/3.13/3.14, while Spark 3.5 declares python_requires=">=3.8" with classifiers listing 3.8/3.9/3.10/3.11. The intersection of the two supported ranges is 3.10 and 3.11; we pick the higher one (3.11.11) so the same image serves both matrix profiles. The old 3.9.21 was below the Spark 4.1 lower bound. Both the Dockerfile and the workflow now document this choice inline so future bumps stay in sync.

The CI Docker image (ghcr.io/${owner}/livy-ci:latest) had to be rebuilt and pushed manually for the PR CI to pick up the new Python before this PR is merged: .github/workflows/build-ci-image.yaml only fires on push to master, so branch/PR runs would otherwise still pull the stale image cached from the previous Dockerfile. Commands used (from macOS Apple Silicon, cross-built for linux/amd64 to match the GitHub Actions runners):

gh auth refresh --scopes write:packages,read:packages
gh auth token | docker login ghcr.io -u <owner> --password-stdin
docker buildx build --platform linux/amd64 \
  -t ghcr.io/<owner>/livy-ci:latest --push \
  dev/docker/livy-dev-base

After this PR merges to master, the workflow will republish the image on any future Dockerfile change automatically, so this manual step is only needed for the bootstrap run.

python-api/setup.py PEP 440 version fix
Version bumped from 1.0.0-SNAPSHOT (Maven-style, not PEP 440 compliant) to 1.0.0.dev0 (the canonical Python "pre-release under active development" form). Modern pip (>= 24) refuses to parse the old value with Invalid version: '1.0.0-SNAPSHOT', which surfaced as WARNING: Error parsing dependencies of livy-python-api on every pip3 install and blocked the validate Python-API requests integration test from resolving its dependencies. The Maven POM version (python-api/pom.xml) stays 1.0.0-SNAPSHOT -- Maven and pip have separate versioning conventions and only the pip-visible metadata needs to change.

How was this patch tested?

• pip3 install ./python-api no longer emits WARNING: Error parsing dependencies of livy-python-api: Invalid version: '1.0.0-SNAPSHOT'; the package installs cleanly on pip 24+.
• Rebuilt the CI Docker image locally with the pyenv 3.11.11 bump and confirmed python3 --version reports Python 3.11.11 inside the container.
• GitHub Actions Integration Tests workflow runs pyenv global 3.11.11 successfully on the -Pspark3 -Pscala-2.12 matrix entry (this commit does not yet add any Spark 4 entry).

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Opus 4.7)

[roczei]

Hi @gyogal and @ArnavBalyan,

This PR is ready for code review. Could you please take a look at this when you have a moment? Any feedback you provide would be greatly appreciated! Thank you!

[gyogal]

The error message in the integration tests (pyenv: version '3.11.11' not installed) is expected, as the livy-dev-base Docker image still contains Python 3.9.21. The image will be updated by the "Build CI images" Github workflow as soon as this PR is merged and subsequent test runs should pass.