[bug][PES][publicservice] reading large result set causes publicservice oom

[v-kkhuang]

Search before asking

• I had searched in the issues and found no similar issues.

Linkis Component

• linkis-commons
• linkis-computation-governance
• linkis-dist
• linkis-engineconn-plugin
• linkis-extensions
• linkis-orchestrator
• linkis-public-enhancements
• linkis-spring-cloud-services
• linkis-web

Description

When reading a very large result set through FsRestfulApi (ps-publicservice), FileSplit.collect() loads all matched rows into memory. Only row-count and column limits existed, with no total byte limit, so a large result set blows up the ps-publicservice heap and causes an OOM that crashes the whole public service.

Steps to reproduce

1. Produce a task result set that is very large (hundreds of MB).
2. Open/read it via the FsRestfulApi result-set endpoint.
3. ps-publicservice runs out of heap and OOMs.

Expected behavior

ps-publicservice should limit the collected bytes and return partial data with a hint instead of OOM.

Your environment

• Linkis version used: dev-2.1.0
• Environment name and version:
  • hadoop-2.7.2 / 3.3.1
  • spark-2.4.3 / 3.4.4
  • scala-2.11.12
  • jdk 1.8

Anything else

The fix adds a collect total-bytes limit (key linkis.filesystem.resultset.collect.max.bytes, default 500m) guarded by a switch (linkis.filesystem.resultset.size.check.enabled, default true). When the limit is exceeded, collect stops early and returns a partial-data hint so users can use the result-set export for the full data.

Are you willing to submit a PR?

• Yes I am willing to submit a PR!