[bug][EC][spark] jackson StreamConstraintsException thrown when spark event log string exceeds default maxStringLength

[v-kkhuang]

Search before asking

• I had searched in the issues and found no similar issues.

Linkis Component

• linkis-commons
• linkis-computation-governance
• linkis-dist
• linkis-engineconn-plugin
• linkis-extensions
• linkis-orchestrator
• linkis-public-enhancements
• linkis-spring-cloud-services
• linkis-web

Description

In Linkis 2.1.0 the spark engineconn is built against Jackson 2.15.0. Jackson 2.15 introduced StreamReadConstraints whose default maxStringLength is 5,000,000 (and maxNestingDepth is 1000). When Spark's EventLoggingListener serializes/deserializes event payloads through json4s-jackson and a single string value exceeds this limit, Jackson throws a StreamConstraintsException: String value length X exceeds the maximum allowed 5,000,000. This aborts event-log handling inside the spark engine and can destabilize the engine session. Jackson 2.15 does not expose a public API to override the global defaults (a global override API was only added in Jackson 2.16), so the constraint cannot be relaxed through normal configuration.

Steps to reproduce

1. Deploy Linkis 2.1.0 and start a spark engineconn (spark 2.4.3 or spark 3.x, both ship Jackson 2.15+).
2. Submit a spark job whose event log produces a string value longer than 5,000,000 characters (for example a very large query plan, a large broadcast variable summary, or a long inlined code/script string).
3. Observe the engine log.

Expected behavior

The spark engineconn should tolerate event-log strings that exceed Jackson's default maxStringLength/maxNestingDepth without throwing StreamConstraintsException, so event logging and the engine session are not interrupted.

Your environment

• Linkis version used: 2.1.0
• Environment name and version:
  • hadoop-2.7.2 (default) / hadoop-3.3.1 (hadoop-3.3 profile)
  • hive-2.3.3
  • spark-2.4.3 (default) / spark-3.4.4 (spark-3 profile)
  • scala-2.11.12 (default) / scala-2.12.18 (spark-3 profile)
  • jackson-databind 2.15.0 (spark-3 profile: 2.14.2)
  • jdk 1.8

Anything else

The fix overrides Jackson's internal StreamReadConstraints.DEFAULT via reflection (the only viable approach on Jackson 2.15, which has no public global-default override API) and exposes two configurable knobs with safe defaults:

• linkis.spark.jackson.maxStringLength (default 10000000)
• linkis.spark.jackson.maxNestingDepth (default 2000)

The override runs once at spark engine session creation and is wrapped in try/catch so that any reflection failure falls back to Jackson's stock defaults without breaking engine startup.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!