[dashboard] Stream job log responses

[bvolpato]

Description

Full job-log requests currently read the entire driver log into memory, serialize it in the job agent, deserialize and serialize it again in JobHead, then buffer it once more in the dashboard subprocess proxy. Large or concurrent requests can exhaust dashboard memory.

This change reads log files in 64 KiB chunks, incrementally JSON-escapes them, and streams the response through the job agent, JobHead, and parent dashboard proxy with backpressure. The public application/json response remains {logs: string}, so JobSubmissionClient.get_job_logs() keeps returning complete logs. Client-side memory remains proportional to returned logs because that stable SDK method returns one string.

Related issues

Fixes #28336.

Duplicate-work check: no open PR implements request-time streaming. Closed PR #61537 capped and truncated large responses; this approach preserves complete output. Open PR #65006 rotates on-disk driver logs and is complementary rather than duplicative. I described these differences on #28336 before opening this draft.

Additional information

Testing:

• .venv/bin/pytest -q python/ray/dashboard/modules/job/tests/test_job_log_storage_client.py python/ray/dashboard/tests/test_utils.py: 4 passed
• PATH=$PWD/.venv/bin:$PATH .venv/bin/pytest -q python/ray/dashboard/modules/job/tests/test_http_job_server.py::test_submit_job_with_exception_in_driver python/ray/dashboard/modules/job/tests/test_http_job_server.py::test_missing_resources: 2 passed
• Ray pytest checker against new test file: passed
• pre-commit run on all changed files: passed
• staged pre-commit run and commit hooks: passed
• npm run build in python/ray/dashboard/client: compiled successfully
• Local head plus Jobs CLI smoke: submitted a successful job producing an 8.39 MiB log, retrieved complete output through ray job logs and direct API, verified valid JSON markers and chunked transfer, then confirmed cluster remained healthy with no dashboard streaming errors
• Three concurrent 1 MiB/s throttled full-log requests all returned valid 8.39 MiB JSON. Observed RSS deltas were approximately +3.4 MiB for dashboard, +2.4 MiB for JobHead, and +0.1 MiB for DashboardAgent
• Post-review local head smoke: successful job produced a 1.99M-character log; CLI and chunked JSON API returned complete output; cluster remained healthy

AI assistance was used.

[gemini-code-assist]

Code Review

This pull request implements log streaming for Ray dashboard jobs, transitioning from loading full logs into memory to streaming them in bounded chunks. It introduces chunked log reading in JobLogStorageClient, updates the job agent and head to support StreamResponse, and adds corresponding unit tests. The review feedback suggests enhancing robustness against encoding issues by explicitly specifying encoding="utf-8" (and errors="replace" when reading) both in the log storage client and the test suite to prevent crashes on non-UTF-8 characters or locales.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0ca9a44f13

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[gemini-code-assist]

Code Review

This pull request implements log streaming for Ray jobs by transitioning from fetching full log payloads to streaming logs in 64KB chunks using aiohttp's StreamResponse. It updates the job agent, job head, and log storage client to support chunked reading and streaming, while also adding robust error handling for streaming cancellations. The feedback suggests two key improvements: first, using the public response.prepared property instead of the internal req.writer.output_size to check if a response has started; second, adding defensive checks when accessing the request writer in the decorator to prevent potential AttributeError or IndexError during testing.

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b3e66f97b4

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{Reviewed by Cursor Bugbot for commit 4f35c0e. Configure here.}

[bvolpato]

Buildkite #71276 failed in an unrelated core test shard, not in dashboard job tests.

• Failing target: //python/ray/tests:test_reference_counting
• Timed-out cases: test_actor_holding_serialized_reference[False-True] and [True-True]
• The target timed out in both attempts after 60–75 seconds. The harness ultimately sent SIGTERM while Python was in garbage collection.
• Worker and actor death messages come from the test's intentional failure path; there is no dashboard or job-log-streaming traceback.
• This PR changes dashboard files only. Premerge build #71257 passed before the two small streaming-timeout overrides in the latest commit.

This points to timing/resource flakiness in the reference-counting test rather than a regression from this PR. Recommended next step: retry the failed premerge job; no code change is indicated.

Failure: https://buildkite.com/ray-project/premerge/builds/71276/list?jid=019fc9d7-13ed-415e-8e88-75e6ebed072d&tab=output