[LIVY-1041] Add Spark 4 support via a new Maven profile - #542
Open
roczei wants to merge 4 commits into
Open
Conversation
## What changes were proposed in this pull request? The Livy unit-test and integration-test suites do not pass on macOS today. The root cause is that several Livy and Spark services bind to the LAN IP returned by InetAddress.getLocalHost, but on macOS the canonical hostname routinely resolves to a loopback-only address (or the LAN IP is otherwise unreachable from a peer process on the same box). As a result the mini-cluster YARN containers and JDBC clients on the same host cannot reach the server side, and tests time out with "RSCClient instance stopped" or hang on "Initial job has not accepted any resources". The changes below are the minimum set required to turn every Livy test suite green on macOS developer machine -- both the unit-test and integration-test runs, on the -Pspark3 profile. Linux CI hosts already resolve the primary hostname to a routable address, so 127.0.0.1 is equally valid there. To avoid hardcoding "127.0.0.1" at every call site, a shared TestUtils.TEST_BIND_HOST constant is introduced in client-common (a @Private Java helper class already on the compile classpath of every affected module) and referenced from every test-scope bind, so future changes can flip a single location. What changed (each item is required for at least one macOS test suite to go green; together they take the whole build to zero failures on macOS): - client-common/TestUtils.java: add the TEST_BIND_HOST = "127.0.0.1" constant with a Javadoc pointing at LIVY-1065. - integration-test MiniCluster.scala, MiniLivyMain: set livy.server.host, livy.server.thrift.bind.host and livy.rsc.rpc.server.address to TEST_BIND_HOST. The RSC key is referenced through the typed RSCConf.Entry.RPC_SERVER_ADDRESS.key() rather than the raw string, matching the rest of the patch. Without these the RSC driver inside the YARN AM container cannot reach the Livy server on macOS and JdbcIT / InteractiveIT fail with "RSCClient instance stopped". - integration-test MiniCluster.scala, MiniCluster.deploy(): set spark.driver.host, spark.yarn.appMasterEnv.SPARK_LOCAL_IP and spark.executorEnv.SPARK_LOCAL_IP to TEST_BIND_HOST in the generated spark-defaults.conf, and set SPARK_LOCAL_IP=TEST_BIND_HOST in the child process environment. SPARK_LOCAL_IP is what the YARN AM driver actually reads on macOS; spark.driver.host on the submit side alone is not honoured in cluster deploy mode. Without these the executor never registers with the driver on macOS and JdbcIT hangs on "Initial job has not accepted any resources". - integration-test MiniCluster.scala, MiniCluster.deploy(): also propagate the host shell PATH into the YARN AM and executor containers (spark.yarn.appMasterEnv.PATH / spark.executorEnv.PATH), so RRunner and PythonRunner can resolve Rscript / python on macOS -- YARN's default sanitised PATH omits /opt/homebrew/bin, which is where Homebrew installs the R and Python interpreters that macOS developers use, and without it RRunner and PythonRunner fail with `error=2, No such file or directory` in the SparkR / PySpark integration tests. - Unit-test fixtures: set RSCConf.Entry.RPC_SERVER_ADDRESS to TEST_BIND_HOST in every fixture that spins up an in-process RSC server -- BaseSessionSpec, ReplDriverSuite, TestSparkClient, TestRpc, ScalaClientTest, BaseInteractiveServletSpec, InteractiveSessionSpec, ThriftServerBaseTest, ThriftSessionTest. Each of these RSC-hosting unit-test classes hangs or fails on macOS without the loopback pin (the child driver JVM times out contacting the server on the machine's routable IP). - client-http HttpClientSpec.scala, LivyConnectionSpec.scala: change the WebServer bind host from "0.0.0.0" / InetAddress.getLocalHost to TEST_BIND_HOST, and switch the URIs the tests build to use server.host so the client and server agree. Drop the now-unused InetAddress import. Fixes the two HTTP-client suites' timeouts on macOS. - thriftserver ThriftServerBaseTest.scala: use the typed RSCConf.Entry.RPC_SERVER_ADDRESS.key() rather than the raw string, matching every other Scala test file in the patch. ## How was this patch tested? - `mvn -Pthriftserver -Pspark3 -Pscala-2.12 -B verify` locally on macOS: all unit-test suites (including livy-repl_2.12, livy-server, livy-thriftserver, livy-thriftserver-session, livy-client-http, livy-scala-api_2.12) pass with these changes; without them the same suites fail on macOS with "RSCClient instance stopped" / connect timeouts to the LAN IP. ## Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Opus 4.7)
## What changes were proposed in this pull request?
Upgrade scalatest 3.0.8 -> 3.2.9 and scalatra 2.6.5 -> 2.8.4. Both
upgrades are prerequisites for Spark 4 / Scala 2.13 support (parent
JIRA LIVY-1041) and are split into a dedicated commit to keep test
migrations separate from core Spark 4 source changes for easier review.
Scalatest 3.0.8 -> 3.2.9:
- Migrate FunSuite/FunSpec/FunSpecLike/FlatSpec test classes to their
3.2 successors AnyFunSuite/AnyFunSpec/AnyFunSpecLike/AnyFlatSpec.
- Move `org.scalatest.Matchers` to
`org.scalatest.matchers.should.Matchers`.
- Add `org.scalatestplus:mockito-3-4_${scala.binary.version}:3.2.9.0`,
since scalatest 3.2 moved `MockitoSugar.mock` into a separate
scalatestplus artifact.
Scalatra 2.6.5 -> 2.8.4:
- Bump `metrics.version` 3.1.0 -> 4.2.19: scalatra 2.8.x's
metrics-servlets pulls in Dropwizard metrics 4.x, and keeping
metrics-core / metrics-healthchecks at 3.1.0 causes a
NoClassDefFoundError for HealthCheckFilter at runtime.
## How was this patch tested?
- Unit tests: `mvn verify -Pspark3 -Pscala-2.12 -Pthriftserver` passes
on JDK 8/17 for all modules with the migrated ScalaTest 3.2 test
suites (matching what was verified as part of the parent LIVY-1041
branch).
- The metrics 4.2.19 bump is tested by the integration test suite
(`mvn integration-test -Pspark3 -Pscala-2.12 -pl
:livy-integration-test`), which previously failed with
`NoClassDefFoundError: com/codahale/metrics/servlets/HealthCheckFilter`
on MiniCluster startup and now passes.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)
## What changes were proposed in this pull request?
Prepares the CI environment and the python-api package metadata for the
upcoming Spark 4 support commit. Both changes are also useful on the
existing Spark 3 build: the PEP 440 fix unblocks `pip3 install
livy-python-api` on modern pip (>= 24), and the Python bump keeps the
CI image on a version supported by both Spark 3.5 and Spark 4.1.
**CI Python bumped to 3.11.11**
dev/docker/livy-dev-base/Dockerfile pyenv Python bumped from 3.9.21 to
3.11.11, and .github/workflows/integration-tests.yaml pins
`pyenv global 3.11.11` explicitly. Rationale (per each Spark release's
python/setup.py): Spark 4.1 declares `python_requires=">=3.10"` with
classifiers listing 3.10/3.11/3.12/3.13/3.14, while Spark 3.5 declares
`python_requires=">=3.8"` with classifiers listing 3.8/3.9/3.10/3.11.
The intersection of the two supported ranges is 3.10 and 3.11; we pick
the higher one (3.11.11) so the same image serves both matrix profiles.
The old 3.9.21 was below the Spark 4.1 lower bound. Both the Dockerfile
and the workflow now document this choice inline so future bumps stay
in sync.
The CI Docker image (`ghcr.io/${owner}/livy-ci:latest`) had to be
rebuilt and pushed manually for the PR CI to pick up the new Python
before this PR is merged: `.github/workflows/build-ci-image.yaml` only
fires on push to master, so branch/PR runs would otherwise still pull
the stale image cached from the previous Dockerfile. Commands used
(from macOS Apple Silicon, cross-built for linux/amd64 to match the
GitHub Actions runners):
gh auth refresh --scopes write:packages,read:packages
gh auth token | docker login ghcr.io -u <owner> --password-stdin
docker buildx build --platform linux/amd64 \
-t ghcr.io/<owner>/livy-ci:latest --push \
dev/docker/livy-dev-base
After this PR merges to master, the workflow will republish the image
on any future Dockerfile change automatically, so this manual step is
only needed for the bootstrap run.
**python-api/setup.py PEP 440 version fix**
Version bumped from `1.0.0-SNAPSHOT` (Maven-style, not PEP 440
compliant) to `1.0.0.dev0` (the canonical Python "pre-release under
active development" form). Modern pip (>= 24) refuses to parse the old
value with `Invalid version: '1.0.0-SNAPSHOT'`, which surfaced as
`WARNING: Error parsing dependencies of livy-python-api` on every
`pip3 install` and blocked the `validate Python-API requests`
integration test from resolving its dependencies. The Maven POM
version (`python-api/pom.xml`) stays `1.0.0-SNAPSHOT` -- Maven and pip
have separate versioning conventions and only the pip-visible metadata
needs to change.
## How was this patch tested?
- `pip3 install ./python-api` no longer emits
`WARNING: Error parsing dependencies of livy-python-api: Invalid
version: '1.0.0-SNAPSHOT'`; the package installs cleanly on pip 24+.
- Rebuilt the CI Docker image locally with the pyenv 3.11.11 bump and
confirmed `python3 --version` reports `Python 3.11.11` inside the
container.
- GitHub Actions Integration Tests workflow runs `pyenv global 3.11.11`
successfully on the -Pspark3 -Pscala-2.12 matrix entry (this commit
does not yet add any Spark 4 entry).
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)
## What changes were proposed in this pull request?
This PR adds support for building and running Apache Livy against
Apache Spark 4.x (pinned to 4.1.2) via two new Maven profiles:
-Pspark4 (Spark 4.1.2 + Hadoop 3.4.1 + Netty 4.2.7 + JDK 17)
-Pscala-2.13 (Scala 2.13.17)
Spark 4 introduced breaking changes that required widespread updates:
JDK 17 minimum, Scala 2.13 only, Netty 4.2.x, a rewritten
Scala REPL, and json4s 4.0.7.
**Build / POM changes**
- pom.xml: new `spark4` and `scala-2.13` profiles; maven-shade-plugin
3.5.0 -> 3.6.2 (bundles an ASM version that understands Java 22
bytecode, needed to shade Jackson 2.18.2's `META-INF/versions/22`
multi-release classes).
- repl/pom.xml: added `json4s-jackson-core` to the shade includes
(json4s 4.0.7 split `JsonMethods` into a separate artifact; missing
it caused a NoClassDefFoundError in PythonInterpreter at runtime).
Also pulls `livy-test-lib` in as a test dependency so the repl tests
can use the new shared `ScalaVersionAware` trait.
- New Scala 2.13 module wrapper POMs: `core/scala-2.13`,
`repl/scala-2.13`, `scala-api/scala-2.13`.
- README.md: documented Spark 4 Python compatibility -- supported
Python versions for Spark 4.1 are 3.10 – 3.14 (added to the
`-Pspark4` Note block)
**Spark 4 / Scala 2.13 source-level fixes**
- `repl/scala-2.13/SparkInterpreter.scala`: new implementation ported
to the Spark 4 SparkILoop (`shell.ILoop`, `PrintWriter`,
`createInterpreter(settings)`, operations via `sparkILoop.intp`,
`ReplCompletion`). To make extra JARs supplied via
`spark.jars.packages` visible to `import ...` we feed those JARs to
the Scala compiler through the `-classpath` argument at Settings
construction time, instead of via a post-init
`IMain.addUrlsToClassPath` call. In Scala 2.13 the latter does not
reliably register URLs with the compiler's `platform.classPath`, so
`import org.codehaus.plexus.util._` fails with `object plexus is not
a member of package org.codehaus`. A new `collectUserJarsClasspath()`
helper walks the context classloader chain to Spark's
`MutableURLClassLoader`, filters out `livy-*` and the wrong-version
`scala-reflect` (same rules as the old `addUrls` path), and joins the
resulting file paths with `File.pathSeparator`. If the chain has no
`MutableURLClassLoader` we log a warning and skip the extra
`-classpath` entry rather than passing an empty argument.
- `AbstractSparkInterpreter.scala`:
* `parseError` skips the leading caret line so `ename` lands on the
"error: ..." message on both Scala 2.12 and 2.13 (2.13 prints the
caret pointer before the message). Implemented with
`lines.indexWhere(_.trim != "^") ... lines.patch(idx, Nil, 1)`.
* New `isEffectivelyEmpty` helper strips block/line comments and
returns success for comment-only inputs (Scala 2.12 accepted them
silently; 2.13 rejects them as compile errors).
- `repl/Session.scala`: SparkR `setJobGroup` match now accepts `"4"`
alongside `"2" | "3"`.
- `server/interactive/InteractiveSession.scala`: `datanucleusJars` and
`mergeHiveSiteAndHiveDeps` now take an extra `scalaVersion`
parameter, adds `case 3 | 4` to the major-version match and replaces
the hard-coded `assembly/target/scala-2.12/jars` with
`assembly/target/scala-$scalaVersion/jars`.
- `server/batch/BatchSession.scala`: pass `--verbose` to spark-submit
under `LivyConf.TEST_MODE` so tests can grep the resolved arguments
out of the child's log.
- `rsc/driver/SparkEntries.java`: new reflection-based
`hiveClassesArePresent` helper that probes
`org.apache.spark.sql.classic.SparkSession$` (Spark 4) first and
falls back to `org.apache.spark.sql.SparkSession$` (Spark 3).
- `utils/SparkProcessBuilder.scala`: new `verbose(v: Boolean)` mutator
and an internal `_verbose` flag that appends `--verbose` to
`spark-submit` when set (used by `BatchSession`).
- `utils/LivySparkUtils.scala`: `(4, 0)` and `(4, 1)` -> `"2.13"`
entries in `sparkScalaVersionMap`; `MAX_VERSION` bumped from
`(3, 6)` to `(4, 2)`.
- `repl/SparkRInterpreter.scala`: log a warning on Spark 4 that
SparkR was deprecated in SPARK-49347 and may be removed in a future
Spark release.
- `scala-api/scalaapi/package.scala`: inline comment documenting that
`Duration.isFinite` is a parameterless def in Scala 2.13 (took an
empty parameter list in 2.12); dropping the parens compiles on both.
- Thriftserver Scala/json4s 4.x updates:
* `session/Get{Columns,Functions,Schemas,Tables}Job.java`:
`scala.collection.JavaConversions.seqAsJavaList` (removed in Scala
2.13) replaced with
`scala.collection.JavaConverters.seqAsJavaListConverter(...).asJava()`.
* `types/DataTypeUtils.scala`: json4s 4.x `parse(input, useBigDecimal)`
requires an `AsJsonInput[T]`; switch to the single-arg String
overload `parse(sparkJson)`.
* `LivyExecuteStatementOperation.scala`: explicit `res.toSeq` since
Scala 2.13 no longer widens a mutable `Buffer` to `Seq` implicitly.
* `ThriftServerSuites.scala`: `LIVY-571` assertion now also accepts
the Spark 4 error string `[SCHEMA_NOT_FOUND] The schema
`spark_catalog`.`invalid_database` cannot be found`.
- `InteractiveSessionServlet.scala`: `logs.asJava` -> `logs.toSeq.asJava`
so Scala 2.13 picks the `Seq[A]` -> `java.util.List` `asJava`
overload instead of erroring on ambiguity.
**Thriftserver libthrift pre-0.16 / 0.16 compatibility**
The Livy Thrift binary CLI service was written against libthrift 0.9.3
(the version pinned in the root pom.xml). Spark 3 keeps a compatible
pre-0.16 libthrift on the runtime classpath (e.g. 0.12.0 in Spark 3.3),
but Spark 4 pulls in libthrift 0.16.0 transitively through spark-hive,
and 0.16 removed four `TThreadPoolServer.Args` builder methods that
Livy relied on: `requestTimeout`, `requestTimeoutUnit`,
`beBackoffSlotLength` and `beBackoffSlotLengthUnit`. Compiling against
0.9.3 but running against 0.16.0 made the mini-cluster Thrift server
abort during startup with a `NoSuchMethodError` on
`TThreadPoolServer$Args.requestTimeout(int)`, which in turn hung
JdbcIT on the SASL handshake. `ThriftBinaryCLIService` now invokes
those four builders reflectively via a small `applyOptionalArg`
helper: they run unchanged on Spark 3 classpaths and are silently
skipped on Spark 4, where they no longer exist. The functional loss
on Spark 4 is limited to the login-timeout / backoff knobs, which
have no equivalent in the newer libthrift API.
**Test fix: Scala 2.13 REPL output shape changes in InteractiveIT**
The Scala 2.13 REPL prints results differently from Scala 2.12 in
four ways that `InteractiveIT` was too strict about; the Spark 4
matrix (Scala 2.13 only) trips all four. Fixed in place:
- Value results: `val res0: Int = 2` (2.13) vs. `res0: Int = 2`
(2.12). Six `verifyResult` regexes (across `basic interactive
session`, `user jars are properly imported ...` and `recover
interactive session`) now accept an optional `val ` prefix.
- Compile errors: `error: not found: value abcde` (2.13) vs.
`<console>:12: error: not found: value abcde` (2.12). The
`<source>:<line>: ` location marker is now optional in the
`verifyError` regex.
- Class definitions: `class Item` (2.13) vs. `defined class Item`
(2.12). The `case class Item(i: Int)` check in `user jars are
properly imported ...` now accepts an optional `defined ` prefix.
- Deprecation-warning header: Scala 2.13 prints a
`warning: 1 deprecation ...` line above the value binding when the
expression uses a deprecated API. `SparkContext.parallelize` is
deprecated on Spark 4, so
`val rdd = sc.parallelize(Array.fill(10){...})` prints the warning
before the `val rdd: ...` line. The `verifyResult` regex now uses
`(?s)(?:warning:.*\n)?(?:val )?rdd.*` to accept the warning prefix
and the DOTALL semantics needed to span the newline.
Both forms match on `-Pscala-2.12` and `-Pscala-2.13`.
**Test fix: Spark 4 SQLContext / DataFrame runtime class in InteractiveIT**
Spark 4 moved the runtime `SQLContext` and `DataFrame` implementations
into `org.apache.spark.sql.classic`; only the compile-time aliases
remain at `org.apache.spark.sql.SQLContext` / `org.apache.spark.sql.DataFrame`.
The Scala 2.13 REPL surfaces the runtime class name, so Spark 4
prints `sql: org.apache.spark.sql.classic.SQLContext = ...` and
`val df: org.apache.spark.sql.classic.DataFrame = ...`, where Spark 3
prints the alias form. Two `InteractiveIT` regexes now allow an
optional `classic.` package qualifier so the same expectations match
both `-Pspark3` and `-Pspark4`:
- `sql: org.apache.spark.sql.(?:classic\.)?SQLContext = ...`
- `(?:val )?df: org.apache.spark.sql.(?:classic\.)?DataFrame`
The old `Pattern.quote("df: org.apache.spark.sql.DataFrame")` literal
was too strict for Spark 4 and caused the `basic interactive session`
case to fail.
**Cross-version test helpers: `ScalaVersionAware` trait**
New `test-lib/.../ScalaVersionAware` trait exposes the small
Scala-2.12 vs 2.13 REPL-output differences as reusable fragments:
- `optionalValPrefix` (`"val "` on 2.13, `""` on 2.12) for exact-match
expected strings.
- `optionalValPrefixRegex` / `optionalDefinedPrefixRegex` /
`optionalWarningPrefixRegex` regex fragments for the corresponding
`verifyResult` regex assertions.
Detection uses the runtime `scala.util.Properties.versionNumberString`
so the same test class works regardless of which Scala the artifact
was compiled against. Mixed into `BaseInterpreterSpec`,
`BaseSessionSpec` and `InteractiveIT`. Downstream specs
(`ScalaInterpreterSpec`, `SharedSessionSpec`, `SparkSessionSpec`,
`InteractiveIT`) rewrote their expected-value strings against these
fragments; some hard-`equal` assertions were relaxed to
`include`/`fullyMatch regex` where the 2.13 REPL adds a deprecation
banner ahead of the result or renumbers `resN` slots differently.
**Server/unit tests: cross-Scala tolerance in `InteractiveSessionSpec`**
The `should get scala version` case now decomposes the JSON result by
hand and asserts `text/plain` equals either `res0: Int = 3\n` or
`val res0: Int = 3\n`, plus separate `status` / `execution_count`
assertions. Straight equality with a decomposed Map broke on the
Spark 4 driver where the 2.13 REPL adds the `val ` prefix.
**Unit tests: `LivySparkUtilsSuite`**
Added `4.0.0` and `4.1.2` to both the supported-version list and the
`defaultSparkScalaVersion` -> `"2.13"` expectations.
**scala-api tests**
- `ScalaClientTestUtils.scala`: `context.sc.parallelize(buffer, ...)`
-> `context.sc.parallelize(buffer.toSeq, ...)` (2.13 no longer
widens a mutable `ArrayBuffer` to `Seq` implicitly for the
`parallelize` signature).
- `ScalaJobHandleTest.scala`: `Duration.Undefined` -> `Duration.Inf`
(2.13's `Await.ready` rejects `Undefined` with "Cannot wait for
Undefined duration of time"); assertion switched from
`verify(mockJobHandle, times(1)).get()` to
`verify(mockJobHandle, atLeastOnce()).isDone`, because 2.13's
`Await.ready` short-circuits when `isCompleted` is already true and
does not call the underlying `ready(atMost)` -- so `.get()` is not
observed on 2.13. `isDone` is exercised by both versions.
**Repl tests: `SparkInterpreterSpec` move**
`SparkInterpreterSpec.scala` moved from `repl/scala-2.12/src/test/...`
into the shared `repl/src/test/scala/...` tree (it was byte-identical
between scala-2.12 and scala-2.13). A new `should skip leading caret
lines in Scala 2.13 error format.` test case was added for the caret-
first `parseError` branch, so the shared spec runs three cases under
both `-Pscala-2.12` and `-Pscala-2.13`.
**CI matrix**
- `.github/workflows/unit-tests.yaml` and `integration-tests.yaml`
converted the flat `maven_profile` x `jdk_path` matrix to an
explicit `include:` list and added a new spark4/JDK-17 entry
(`-Pscala-2.13 -Pspark4` on `/usr/lib/jvm/java-17-openjdk-amd64`).
## How was this patch tested?
- Unit tests: `mvn verify -Pspark4 -Pscala-2.13 -Pthriftserver` passes
all 19 modules on JDK 17 (macOS aarch64). Verified modules include
livy-rsc (40 tests), livy-repl_2.13 (87 tests), livy-server
(205 tests), livy-client-http (17 tests), livy-scala-api_2.13
(17 tests), and the thriftserver integration suite (27 tests:
HttpThriftServerSuite + BinaryThriftServerSuite). The
`SparkInterpreterSpec` move added one new caret-first `parseError`
test case; the shared spec now has 3 cases and runs identically
under both scala-2.12 and scala-2.13.
- Integration tests (`mvn integration-test -Pspark3 -Pscala-2.12 -pl
:livy-integration-test`) pass.
- GitHub Actions CI runs both spark3 and spark4 matrix entries for
unit and integration test workflows.
## Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)
Contributor
Author
|
The integration test failed because Python 3.11.11 is missing from the test environment: This will be resolved once #540 is merged and the Docker image is updated. Related comment from @gyogal: #540 (comment) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR adds support for building and running Apache Livy against Apache Spark 4.x (pinned to 4.1.2) via two new Maven profiles:
-Pspark4 (Spark 4.1.2 + Hadoop 3.4.1 + Netty 4.2.7 + JDK 17)
-Pscala-2.13 (Scala 2.13.17)
Spark 4 introduced breaking changes that required widespread updates:
JDK 17 minimum, Scala 2.13 only, Netty 4.2.x, rewritten
Scala REPL, json4s 4.0.7 and new Scalatra 2.8.4 / ScalaTest 3.2 deps.
The PR is split into four commits so each self-contained prerequisite
can be reviewed on its own JIRA. Reviewers can look at them individually
in the order below; each is functional on its own on top of master.
Commit 1 — [LIVY-1065] Ensure all tests pass on macOS
On hosts (macOS, some CI containers) whose canonical hostname resolves
to a loopback-only address, several Livy and Spark services bound to
the LAN IP returned by
InetAddress.getLocalHost, which the mini-clusterYARN containers and JDBC clients on the same host could not reach.
This commit turns every Livy unit-test and integration-test suite green
on macOS, and is a no-op on Linux CI (where the canonical hostname
already resolves to a routable address, so
127.0.0.1is equally valid).To avoid hard-coding
127.0.0.1at every call site, a sharedTestUtils.TEST_BIND_HOSTconstant is added inclient-common(a@PrivateJava helper already on the compile classpath of everyaffected module) and referenced from every test-scope bind.
Five independent bind points get the same loopback override:
livy.server.host= TEST_BIND_HOSTlivy.server.thrift.bind.host= TEST_BIND_HOST(without it JdbcIT connected to
jdbc:hive2://<lanIp>:10090andtimed out during the SASL handshake).
livy.rsc.rpc.server.address= TEST_BIND_HOST(RSCConf.findLocalAddress otherwise picks the LAN IP, so the Spark
driver running inside the YARN AM container could not reach the Livy
RSC server and JdbcIT failed with "RSCClient instance stopped" caused
by ConnectTimeoutException on
/<lanIp>:10000).spark.driver.host= TEST_BIND_HOST inMiniCluster.deploy()'s generated
spark-defaults.conf. This alone isnot enough in YARN cluster mode because the driver runs inside the AM
container and reads its bind address from
SPARK_LOCAL_IPthere.spark.yarn.appMasterEnv.SPARK_LOCAL_IPandspark.executorEnv.SPARK_LOCAL_IP= TEST_BIND_HOST (without thesethe driver AM still advertised
spark://YarnAM@<lanIp>:<port>fromInetAddress.getLocalHostand the executor container neverregistered, so JdbcIT hung on "Initial job has not accepted any
resources").
Test / CI PATH propagation for native runners.
MiniCluster.deploy()also propagates the host shell PATH into the YARN AM and executor
containers via
spark.yarn.appMasterEnv.PATHandspark.executorEnv.PATH.YARN's default sanitised PATH otherwise omits
/opt/homebrew/binonmacOS and any user-managed pyenv/rbenv shims on CI runners, which
causes Spark's RRunner (SparkRIT) and PythonRunner (PySparkIT) to fail
with
java.io.IOException: Cannot run program "Rscript"/"python": error=2, No such file or directory.Unit-test fixture updates. Every fixture that spins up an
in-process RSC server now pins
RSCConf.Entry.RPC_SERVER_ADDRESStoTEST_BIND_HOST —
BaseSessionSpec,ReplDriverSuite,TestSparkClient,TestRpc,ScalaClientTest,BaseInteractiveServletSpec,InteractiveSessionSpec,ThriftServerBaseTest,ThriftSessionTest.HttpClientSpec/LivyConnectionSpecswitch the WebServer bind hostfrom
0.0.0.0/InetAddress.getLocalHostto TEST_BIND_HOST so theclient and server URIs agree.
Commit 2 — [LIVY-1066] Upgrade scalatest to 3.2.9 and scalatra to 2.8.4
Both upgrades are prerequisites for Spark 4 / Scala 2.13 support and
are split into a dedicated commit to keep test migrations separate
from core Spark 4 source changes for easier review.
scalatest 3.0.8 → 3.2.9
FunSuite/FunSpec/FunSpecLike/FlatSpectest classes totheir 3.2 successors
AnyFunSuite/AnyFunSpec/AnyFunSpecLike/AnyFlatSpec.org.scalatest.Matcherstoorg.scalatest.matchers.should.Matchers.org.scalatestplus:mockito-3-4_${scala.binary.version}:3.2.9.0,since scalatest 3.2 moved
MockitoSugar.mockinto a separatescalatestplus artifact.
scalatra 2.6.5 → 2.8.4
metrics.version3.1.0 → 4.2.19: scalatra 2.8.x'smetrics-servlets pulls in Dropwizard metrics 4.x, and keeping
metrics-core/metrics-healthchecksat 3.1.0 causes aNoClassDefFoundErrorforHealthCheckFilterat runtime onMiniCluster startup.
Commit 3 — [LIVY-1067] Bump CI Python to 3.11.11 and fix python-api PEP 440 version
Prepares the CI environment and the python-api package metadata for the
upcoming Spark 4 support commit. Both changes are also useful on the
existing Spark 3 build: the PEP 440 fix unblocks
pip3 install livy-python-apion modern pip (>= 24), and the Python bump keeps theCI image on a version supported by both Spark 3.5 and Spark 4.1.
CI Python bumped to 3.11.11
dev/docker/livy-dev-base/Dockerfilepyenv Python bumped from 3.9.21to 3.11.11, and
.github/workflows/integration-tests.yamlpinspyenv global 3.11.11explicitly. Rationale (per each Spark release'spython/setup.py): Spark 4.1 declarespython_requires=">=3.10"withclassifiers listing 3.10/3.11/3.12/3.13/3.14, while Spark 3.5 declares
python_requires=">=3.8"with classifiers listing 3.8/3.9/3.10/3.11.The intersection of the two supported ranges is 3.10 and 3.11; we pick
the higher one (3.11.11) so the same image serves both matrix profiles.
The old 3.9.21 was below the Spark 4.1 lower bound. Both the Dockerfile
and the workflow now document this choice inline so future bumps stay
in sync.
The CI Docker image (
ghcr.io/${owner}/livy-ci:latest) had to berebuilt and pushed manually for the PR CI to pick up the new Python
before this PR is merged:
.github/workflows/build-ci-image.yamlonlyfires on push to master, so branch/PR runs would otherwise still pull
the stale image cached from the previous Dockerfile. Commands used
(from macOS Apple Silicon, cross-built for linux/amd64 to match the
GitHub Actions runners):
After this PR merges to master, the workflow will republish the image
on any future Dockerfile change automatically, so this manual step is
only needed for the bootstrap run.
python-api/setup.py PEP 440 version fix
Version bumped from
1.0.0-SNAPSHOT(Maven-style, not PEP 440compliant) to
1.0.0.dev0(the canonical Python "pre-release underactive development" form). Modern pip (>= 24) refuses to parse the old
value with
Invalid version: '1.0.0-SNAPSHOT', which surfaced asWARNING: Error parsing dependencies of livy-python-apion everypip3 installand blocked thevalidate Python-API requestsintegration test from resolving its dependencies. The Maven POM
version (
python-api/pom.xml) stays1.0.0-SNAPSHOT— Maven and piphave separate versioning conventions and only the pip-visible metadata
needs to change.
Commit 4 — [LIVY-1041] Add Spark 4 support via a new Maven profile
Build / POM changes
pom.xml: newspark4andscala-2.13profiles;maven-shade-plugin3.5.0 → 3.6.2 (Java 22 bytecode support for Jackson 2.18.2
multi-release jars).
repl/pom.xml: addedjson4s-jackson-coreto shade includes (json4s4.0.7 split
JsonMethodsinto a separate artifact; missing it causeda
NoClassDefFoundErrorinPythonInterpreterat runtime).core/scala-2.13,repl/scala-2.13,scala-api/scala-2.13.Spark 4 / Scala 2.13 source-level fixes
repl/scala-2.13: newSparkInterpreter.scalaported to the Spark 4SparkILoop(shell.ILoop,PrintWriter,createInterpreter(settings),operations via
sparkILoop.intp,ReplCompletion). To make extra JARssupplied via
spark.jars.packagesvisible toimport ...we nowfeed those JARs to the Scala compiler through the
-classpathargument at
Settingsconstruction time, instead of trying to injectthem after the fact via
IMain.addUrlsToClassPath. In Scala 2.13 thelatter no longer registers URLs with the compiler's
platform.classPathin a way
import ...can resolve (global.extendCompilerClassPathruns, but
import org.codehaus.plexus.util._still fails withobject plexus is not a member of package org.codehaus). A newcollectUserJarsClasspath()helper walks the context classloaderchain to Spark's
org.apache.spark.util.MutableURLClassLoader, filtersout
livy-*and the wrong-version scala-reflect (same rules the oldaddUrls path used), and joins the resulting file paths with
File.pathSeparator. If the chain does not contain aMutableURLClassLoaderwe log a warning and skip the extra-classpathentry rather than passing an empty argument. This is the same
mechanism Spark 4's own
spark-shelluses inrepl/src/main/scala/org/apache/spark/repl/Main.scala.AbstractSparkInterpreter:parseErrorskips leading caret lines(Scala 2.13 REPL format);
isEffectivelyEmptyreturns success forcomment-only inputs (Scala 2.13 rejects them as errors).
Session.scala: SparkR job-group match accepts "4".InteractiveSession:datanucleusJarscase 4; parameterisescala-2.12 hard-coded path;
Future.onSuccess/onFailureremovedin Scala 2.13.
SparkEntries.java: reflection-basedhiveClassesArePresenthelperfor both Spark 3 (
SparkSession$) and Spark 4 (classic.SparkSession$).SparkProcessBuilder:verbose(boolean)mutator for test-mode--verbose.LivySparkUtils: Spark 4.0/4.1 → Scala 2.13 version map entries.Thriftserver:
JavaConversions(removed in 2.13) →JavaConverters;json4s 4.x parse API;
ListBuffer.toSeq; Scala 2.13 error messagesin LIVY-571 assertions.
Various Scala 2.13 collection widening fixes:
toSeq,IndexedSeq.empty,asScala.toSeqacrossSparkYarnApp,ZooKeeperManager,InteractiveSessionServlet.Thriftserver libthrift 0.9 / 0.16 compatibility
The Livy Thrift binary CLI service was written against libthrift 0.9.3
(shipped with Hive 3 and pinned by the Spark 3 profile). Spark 4 pulls
in libthrift 0.16.0 transitively through spark-hive, and 0.16 removed
four
TThreadPoolServer.Argsbuilder methods that Livy relied on:requestTimeout,requestTimeoutUnit,beBackoffSlotLengthandbeBackoffSlotLengthUnit. Compiling against 0.9 but running against0.16 made the mini-cluster Thrift server abort during startup with a
NoSuchMethodErroronTThreadPoolServer$Args.requestTimeout(int),which in turn hung JdbcIT on the SASL handshake.
ThriftBinaryCLIServicenow invokes those four builders reflectively via a small
applyOptionalArghelper: they run unchanged on the Spark 3 classpathand are silently skipped on Spark 4, where they no longer exist. The
functional loss on Spark 4 is limited to the login-timeout / backoff
knobs, which have no equivalent in the newer libthrift API.
Test fix: Scala 2.13 REPL output shape changes in InteractiveIT
The Scala 2.13 REPL prints results differently from Scala 2.12 in four
ways that InteractiveIT was too strict about; the Spark 4 matrix
(Scala 2.13 only) trips all four. Fixed in place:
val res0: Int = 2(2.13) vs.res0: Int = 2(2.12).Six
verifyResultregexes (acrossbasic interactive session,user jars are properly imported ...andrecover interactive session) now accept an optionalvalprefix.error: not found: value abcde(2.13) vs.<console>:12: error: not found: value abcde(2.12). The<console>:<line>:location marker is now optional in theverifyErrorregex.class Item(2.13) vs.defined class Item(2.12). The
case class Item(i: Int)check inuser jars are properly imported ...now accepts an optionaldefinedprefix.warning: 1 deprecation ...line above the value binding when theexpression uses a deprecated API.
SparkContext.parallelizeisdeprecated on Spark 4, so
val rdd = sc.parallelize(Array.fill(10){...})prints the warningbefore the
val rdd: ...line. The verifyResult regex now uses(?s)(?:warning:.*\n)?(?:val )?rdd.*to accept the warningprefix and the DOTALL semantics needed to span the newline.
Both forms match on -Pscala-2.12 and -Pscala-2.13.
Test fix: Spark 4 SQLContext / DataFrame runtime class in InteractiveIT
Spark 4 moved the runtime
SQLContextandDataFrameimplementationsinto
org.apache.spark.sql.classic; only the compile-time aliasesremain at
org.apache.spark.sql.SQLContext/org.apache.spark.sql.DataFrame.The Scala 2.13 REPL surfaces the runtime class name, so Spark 4 prints
sql: org.apache.spark.sql.classic.SQLContext = org.apache.spark.sql.classic.SQLContext@...and
val df: org.apache.spark.sql.classic.DataFrame = ..., where Spark3 prints the alias form. Two InteractiveIT regexes now allow an
optional
classic.package qualifier so the same expectations matchboth -Pspark3 and -Pspark4:
sql: org.apache.spark.sql.(?:classic\.)?SQLContext = ...(?:val )?df: org.apache.spark.sql.(?:classic\.)?DataFrameThe old
Pattern.quote("df: org.apache.spark.sql.DataFrame")literalwas too strict for Spark 4 and caused the
basic interactive sessionInteractiveIT case to fail.
CI matrix
How was this patch tested?
mvn verify -Pspark4 -Pscala-2.13 -Pthriftserverpassesall 19 modules on JDK 17 (macOS aarch64). Verified modules include
livy-rsc (40 tests), livy-repl_2.13 (87 tests), livy-server (205 tests),
livy-client-http (17 tests), livy-scala-api_2.13 (17 tests), and the
thriftserver integration suite (27 tests: HttpThriftServerSuite +
BinaryThriftServerSuite).
mvn verify -Pspark3 -Pscala-2.12) isunaffected and its test suite remains green.
mvn integration-test -Pspark3 -Pscala-2.12 -pl :livy-integration-test) pass once the metrics.version bump resolvesthe HealthCheckFilter NoClassDefFoundError on MiniCluster startup.
and integration test workflows.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Opus 4.7)