[dashboard] Configurable defaults + UI dialogs for py-spy/memray profiling params

[marwan116]

Why are these changes needed?

The Ray Dashboard's profiling actions — Stack Trace (py-spy), CPU Flame Graph (py-spy), and
Memory Profiling (memray) — expose several parameters (native, subprocesses, idle,
leaks, trace_python_allocators, duration, format) whose defaults were hardcoded. The
only way to change one (e.g. to include native C/C++ frames in a stack trace) was to hand-edit the
request URL's query string, and there was no way for a cluster operator to change a default
cluster-wide. This came up from a user asking whether the default value of native can be
configured — today it can't.

This PR makes every profiling parameter's default configurable and gives the dashboard UI first-class
controls for them.

Backend

• New RAY_DASHBOARD_PROFILING_* environment variables, read on the Ray head node, let an operator
  set each default. Parse sites in reporter_head fall back to these constants only when the request
  omits the parameter; an explicit query parameter still wins, so behavior is unchanged unless an
  operator opts in.
• duration and format are split into CPU and memory variants, because py-spy (CPU:
  flamegraph/raw/speedscope) and memray (memory: flamegraph/table) accept different
  valid format sets and different duration defaults (5s vs 10s). A single shared default would break
  CPU requests. format defaults are validated against the correct set at load time, falling back
  to flamegraph on a bad value.
• duration validation now returns HTTP 400, not 500. The endpoints previously raised a bare
  ValueError for an over-long duration, which surfaced as an uncaught HTTP 500. _query_duration
  now raises aiohttp.web.HTTPBadRequest for a non-integer or out-of-range value.
• The accepted duration range is [1, RAY_DASHBOARD_PROFILING_MAX_DURATION_S], defaulting to
  the 60s cap the endpoints have always enforced. Operators raise or lower it per cluster. Configured
  duration defaults are clamped into that range at load time — with a warning naming the offending
  env var — so a misconfigured head node can never bypass the endpoint cap.
• native remains Linux-only regardless of the default (existing guard in profile_manager).
• /api/profiling_enabled now also returns the configured defaults so the UI can seed its dialogs,
  including maxDuration, so the duration field validates against the real cap instead of a
  hardcoded one.
• The valid format sets live in ray_constants and are now read by both reporter_head (to
  validate the configured default) and profile_manager (to validate the request), so the accepted
  formats are declared in exactly one place.

New env vars, all documented in profiling.md (10 total):
RAY_DASHBOARD_PROFILING_NATIVE_DEFAULT, _SUBPROCESSES_DEFAULT, _IDLE_DEFAULT, _LEAKS_DEFAULT,
_TRACE_PYTHON_ALLOCATORS_DEFAULT, _CPU_DURATION_DEFAULT, _MEMORY_DURATION_DEFAULT,
_CPU_FORMAT_DEFAULT, _MEMORY_FORMAT_DEFAULT, RAY_DASHBOARD_PROFILING_MAX_DURATION_S.

Frontend

• The Stack Trace and CPU Flame Graph buttons (worker/actor/node/job and task variants) were
  bare links that hardcoded native=0, which also overrode the operator's configured default.
  They are now parameter dialogs — the same UX as the existing Memory Profiling dialog — so users set
  parameters in the UI instead of editing URLs.
• All three profiling dialogs render one shared ProfilingParamsDialog component and seed their
  initial values from the defaults returned by /api/profiling_enabled. The hardcoded native=0 is
  gone; format options are profiler-correct; the duration field validates against the backend's
  reported maxDuration and disables submit with an inline error when out of range.

Not a duplicate

Searched open PRs before starting — no PR addresses this area:

gh pr list --repo ray-project/ray --state open --search "dashboard profiling native default"  # No Pull Requests
gh pr list --repo ray-project/ray --state open --search "RAY_DASHBOARD_PROFILING"              # No Pull Requests

There is no existing mechanism for configuring dashboard profiling defaults; RAY_DASHBOARD_ENABLE_PROFILING
only gates the feature on/off.

Related issues

Checks

AI assistance (Claude Code) was used to author this change; every changed line was reviewed by the
submitter.

Backend — verified locally:

pytest python/ray/dashboard/modules/reporter/tests/test_reporter.py \
  -k "query or clamp or validated_profiling_format" -v
#   30 passed

pytest python/ray/dashboard/modules/reporter/tests/test_reporter.py -k "profiling_enabled" -v
#   1 passed

pytest python/ray/dashboard/modules/reporter/tests/test_profile_manager.py -v
#   13 passed

Tests in test_reporter.py: test_query_flag, test_query_duration_valid,
test_query_duration_invalid_raises_bad_request, test_clamp_profiling_duration,
test_clamp_profiling_duration_warns_and_names_the_env_var,
test_clamp_profiling_duration_silent_when_in_range, test_validated_profiling_format,
test_validated_profiling_format_absent_uses_fallback,
test_profiling_enabled_endpoint_returns_defaults. These cover that out-of-range durations surface
as HTTP 400 rather than a bare ValueError, that a misconfigured env default is clamped with a
warning, and that the CPU/memory format split rejects speedscope for memray and table for py-spy.

New TestProfilingFormatValidation in test_profile_manager.py asserts both profilers honor the
shared format sets — every declared CPU format reaches the py-spy command, every declared memory
format maps to its memray subcommand, and each profiler rejects the other's formats. It mocks out
subprocess execution, so unlike the live TestMemoryProfiling suite it also runs on macOS.

End-to-end against a live head node (RAY_DASHBOARD_ENABLE_PROFILING=1):

GET /api/profiling_enabled          -> maxDuration=60, cpuDuration=5, memoryDuration=10
GET /worker/cpu_profile?duration=60  -> passes validation
GET /worker/cpu_profile?duration=61  -> HTTP 400  "duration must be between 1 and 60 seconds, got: 61"
GET /worker/cpu_profile?duration=0   -> HTTP 400
GET /worker/cpu_profile?duration=abc -> HTTP 400  "duration query parameter must be an integer"
RAY_DASHBOARD_PROFILING_MAX_DURATION_S=120 -> maxDuration=120

Frontend — verified locally:

cd python/ray/dashboard/client
./node_modules/.bin/tsc --noEmit -p tsconfig.json          # 0 errors
CI=true npx react-scripts test --testPathPattern=ProfilingLink --watchAll=false
#   Test Suites: 1 passed, 1 total
#   Tests:       11 passed, 11 total
./node_modules/.bin/eslint src/common/ProfilingLink*.tsx   # clean
./node_modules/.bin/prettier --check src/common/ProfilingLink*.tsx  # clean

The frontend tests assert, among other things, that the worker Stack Trace link produces
worker/traceback?...&native=1&subprocesses=0 (reflecting a configured native default), that it
never emits a hardcoded native=0, and that an out-of-range duration disables submit with an
inline error.

pre-commit run --files <changed> passes (ruff, black, pydoclint, docstyle, import-order, eslint,
semgrep).

Notes for reviewers

• All new endpoints/parameters stay behind the existing RAY_DASHBOARD_ENABLE_PROFILING security
  gate — unchanged.
• jax_profile is deliberately not migrated to the shared _query_duration helper. It has never
  enforced a duration cap, and routing it through the shared validator would newly reject durations
  above the cap for JAX users — a behavior change unrelated to this PR's goal. Its docstring is
  accurate as-is.
• ProfilingLink.tsx keeps its own copy of the format lists for the dropdown options, since the
  frontend can't import the Python constants. Plumbing them through /api/profiling_enabled would be
  a reasonable follow-up.

[gemini-code-assist]

Code Review

This pull request introduces operator-configurable environment variables to set cluster-wide defaults for profiling parameters (such as duration, format, and various flags) on the Ray head node, updating both the backend and frontend to respect these defaults. The review feedback highlights critical validation and accessibility improvements: the backend endpoints in reporter_head.py should properly validate that profiling durations are between 1 and 60 seconds and raise clean HTTP 400 errors instead of uncaught ValueErrors (which cause HTTP 500s). On the frontend, the duration input field in ProfilingLink.tsx needs visual validation feedback, and invalid HTML nesting of a <Link> inside a <Button> should be resolved by rendering the button directly as an anchor.

[marwan116]

Thanks for the review @gemini-code-assist. Addressed all five comments in 5a037a6:

• Duration validation (3 endpoints): consolidated parsing + validation into one _query_duration helper that raises HTTPBadRequest (400) for non-integer or out-of-range values and enforces a 1–60s bound. This fixes the bare ValueError→500 on the CPU endpoints and adds the upper bound the memory endpoint was missing.
• Frontend duration field: inline error/helperText and the submit action is disabled while the value is out of range.
• Invalid nested interactive elements: the submit Button now renders as the anchor itself (component="a") instead of wrapping a <Link>.

Added tests for the duration bounds (backend) and the invalid-duration disabled/error state (frontend). Frontend verified locally: tsc --noEmit + 10 Jest tests pass, prettier/eslint clean. Backend: pre-commit clean; _query_duration unit-tested (bounds + 400 behavior).

[gemini-code-assist]

The consolidation of parsing and validation into the _query_duration helper, along with the added frontend validation and test coverage, addresses the identified issues regarding parameter bounds and error handling. The approach of using the component="a" prop for the submit button is a standard way to avoid invalid nested interactive elements in React. These changes appear to correctly implement the requested configuration surface for profiling parameters.

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7333e62. Configure here.}

[carolynwang]

lgtm thanks!

[MortalHappiness]

Please also update your stale PR descriptions. Thanks.

[marwan116]

Thanks for the review @MortalHappiness — all four comments addressed in efe7a93, replies inline.

Description is refreshed. It had drifted in a few ways:

• Said "9 env vars total" and omitted RAY_DASHBOARD_PROFILING_MAX_DURATION_S. It's 10.
• Referenced _query_int / test_query_int / test_query_int_invalid_raises_value_error, which were renamed to _query_duration / test_query_duration_valid / test_query_duration_invalid_raises_bad_request.
• Quoted a stale frontend test count (9; it's 11 now).
• Never mentioned three behaviors that had landed since it was written: duration validation moving from a bare ValueError (HTTP 500) to HTTPBadRequest (400), maxDuration being exposed through /api/profiling_enabled so the UI validates against the real cap, and configured duration defaults being clamped at load time.

The Checks section now has re-run numbers (backend 30 + 1 + 13 passed, frontend 11 passed, tsc clean, pre-commit clean) plus the live end-to-end results for the cap.

[MortalHappiness]

LGTM for Python changes. I didn't review frontend and doc changes.

[dstrodtman]

Docs style review (Anyscale docs team)

Run by Douglas Strodtman on the Anyscale docs team, with Claude Code assisting. I read every comment below and stand behind it.

Scope: style, grammar, and Ray docs conventions on the two files under doc/, checked against Ray's documentation style guide. I'm not a code owner here. Everything technical stays with @carolynwang and @MortalHappiness, and I noticed @MortalHappiness's approval explicitly excluded the doc changes, which is why I picked them up.

Nice work on the reference table. Cross-checking every row against ray_constants.py at 2f03aa9, all ten variable names, all ten defaults, the [1, 60] bound, the clamping behavior, the HTTP 400 on an out-of-range explicit duration, and both format sets are accurate. That's a better hit rate than most env-var tables in the docs, so the comments below are almost entirely prose mechanics.

Two things worth a look beyond style:

1. The native row's "only takes effect on Linux" may be scoped too narrowly. RAY_DASHBOARD_PROFILING_NATIVE_DEFAULT feeds native on the memory endpoint too (reporter_head.py, memory_profile), and your own code comment says memray native is cross-platform while py-spy only appends --native on Linux. If that's right, the Linux caveat holds for stack trace and CPU profiling but not for memory profiling, and the table currently reads as though it applies to all three. Flagging rather than suggesting a fix, since you know the profiler behavior and I'm reading it off your comment. Details inline.
2. The PR title promises UI dialogs, and the docs only cover the env vars. A reader hitting the dashboard sees new parameter dialogs that no page describes. Is documenting them in scope here, or a deliberate follow-up? Not a blocker from me either way, just want it to be a decision rather than an oversight.

One out-of-diff note: profiling.md has no myst.html_meta.description front matter, which the style guide asks for on every page. Pre-existing and above the diff, so there's nothing to click, but it's a cheap add while you're in the file.

Happy to give this a docs-side stamp once you've picked through the comments. Posting as a comment, not a change request.

[dstrodtman]

Heading level and placement. This is an ### under ## Enabling dashboard profiling, which makes configuring defaults a subtask of enabling profiling. They read more like siblings to me: enabling is a security gate you pass once, configuring defaults is separate operator tuning, and this section sits right before ## CPU profiling. Worth considering promoting it to ## Configuring profiling defaults. Your call, and I can see the argument that both are head-node env vars so grouping them helps.

On the wording: the style guide asks for imperative task headings ("Configure profiling defaults"), but the H2 above it is a gerund, and internal consistency on the page wins over a global sweep. Leaving it as-is is fine.

[dstrodtman]

Two parenthetical asides in one paragraph, and the style guide asks you to restructure rather than parenthesize. Folding the first into the sentence and promoting the second to its own sentence also gives the table a cleaner "the following" lead-in:

Suggested change

	Each profiling request (stack trace, CPU flame graph, memory profile) accepts several parameters. When a request omits a parameter, the value falls back to a cluster-wide default. Set these environment variables on the Ray head node to change the defaults for every request that doesn't specify the parameter explicitly (an explicit query parameter always takes precedence).
	Stack trace, CPU flame graph, and memory profile requests each accept several parameters. When a request omits a parameter, its value falls back to a cluster-wide default. Set the following environment variables on the Ray head node to change those defaults. An explicit query parameter always takes precedence.

[dstrodtman]

Semicolon in prose. The style guide asks you to split instead:

Suggested change

	- Include native (C/C++) stack frames. Higher overhead; only takes effect on Linux.
	- Include native (C/C++) stack frames. Adds significant overhead. Only takes effect on Linux.

Separately, and this is the technical question from the review body rather than a style point: I don't think "only takes effect on Linux" is true for all three request types. memory_profile in reporter_head.py also defaults its native flag from this variable, and your comment on the pyspy_native_supported field says memray native is cross-platform and only py-spy is Linux-gated. If so, a reader who sets this to 1 on macOS gets no change to stack traces or CPU profiles but does get native frames in memory profiles, which this row tells them not to expect.

I've deliberately kept the suggestion above to the semicolon and left the Linux claim exactly as you wrote it, because you know the profiler behavior and I don't want to restyle a scope claim into something more confident than I can verify. If the split is real, something like "Only takes effect on Linux for stack trace and CPU profiling. memray honors it on every platform." would cover it.

[dstrodtman]

The style guide asks you to spell out separators in prose, so X / Y becomes "X or Y":

Suggested change

	- Include off-CPU / sleeping threads (CPU profiling only).
	- Include off-CPU or sleeping threads (CPU profiling only).

[dstrodtman]

This is the one cell doing four jobs, and it has a semicolon plus an abbreviated unit. Splitting it into sentences and spelling out the unit:

Suggested change

	- Maximum accepted profiling `duration` in seconds. A profile blocks the request for its whole duration, so it's capped rather than open-ended; raise or lower it per cluster. The minimum is always 1s. Explicit `duration` query values above this return HTTP 400.
	- Maximum accepted profiling `duration` in seconds. A profile blocks the request for its whole duration, so Ray caps it rather than leaving it open-ended. Raise or lower it per cluster. The minimum is always 1 second. An explicit `duration` query value above this maximum returns HTTP 400.

I verified each claim in there against _profiling_duration_cap, _clamp_profiling_duration, and _query_duration, so this is a rewording only, not a change in meaning.

[dstrodtman]

Small one: add the conjunction before the last item in each set, per the guide's Oxford comma rule.

Suggested change

	- Output format for CPU profiling. One of `flamegraph`, `raw`, `speedscope`.
	- Output format for CPU profiling. One of `flamegraph`, `raw`, or `speedscope`.

[dstrodtman]

Same here:

Suggested change

	- Output format for memory profiling. One of `flamegraph`, `table`.
	- Output format for memory profiling. One of `flamegraph` or `table`.

[dstrodtman]

Three things in this paragraph:

• "prefer leaving it off and enabling it only when" is hedged. The guide asks you to turn hedging into direct advice, so make it an instruction and give the reason.
• "(Linux only)" repeats the table row two lines above.
• The overhead warning now appears three times across this PR: this paragraph, the table's native row, and the new sentence in debug-hangs.rst. Once in the table plus once in the prose that a stack-trace reader actually lands on is enough.

Suggested change

	For example, to make native frames the default for stack traces across the cluster (Linux only), set `RAY_DASHBOARD_PROFILING_NATIVE_DEFAULT=1` on the head node. Because `native` significantly increases profiling overhead, prefer leaving it off and enabling it only when sampling the Python layer alone isn't enough.
	For example, to make native frames the default for stack traces across the cluster, set `RAY_DASHBOARD_PROFILING_NATIVE_DEFAULT=1` on the head node. Enable it only when sampling the Python layer alone isn't enough, because native frames significantly increase profiling overhead.

[dstrodtman]

Two style points on the added sentences, plus one that isn't yours.

The (see :ref:...) parenthetical is the pattern the style guide asks you to restructure, and "prefer enabling it only when" is hedged where a direct instruction reads better. Moving the cross-reference to its own sentence at the end also makes it the obvious next click. I kept the existing hard wrap so the diff stays tight, and left the (only supported on Linux) parenthetical on line 14 alone since it predates this PR.

Suggested change

	trace is shown. To show native code frames, set the URL parameter ``native=1`` (only supported on Linux). To make native
	frames the default for every stack trace on the cluster, set the ``RAY_DASHBOARD_PROFILING_NATIVE_DEFAULT=1`` environment
	variable on the Ray head node (see :ref:`Configuring profiling defaults <profiling-defaults>`). Because native frames add
	significant profiling overhead, prefer enabling it only when sampling the Python layer alone isn't enough.
	trace is shown. To show native code frames, set the URL parameter ``native=1`` (only supported on Linux). To make native
	frames the default for every stack trace on the cluster, set the ``RAY_DASHBOARD_PROFILING_NATIVE_DEFAULT=1`` environment
	variable on the Ray head node. Native frames add significant profiling overhead, so enable them only when sampling the
	Python layer alone isn't enough. See :ref:`Configuring profiling defaults <profiling-defaults>`.

Not from your diff, so there's nothing to click, but line 13 just above reads "Clicking "Stack Trace" will return the current stack trace sample" and the guide asks for present tense over "will" for things that are always true. "returns" would fix it if you feel like touching the line while you're here.

[dstrodtman]

Stamping with small doc feedback items.

[MortalHappiness]

Hi @marwan116 Can you check if this test failure related to your changes? https://buildkite.com/ray-project/premerge/builds/71537#019fd9d4-470c-4303-ac7a-b236c4d40838

I've synced with the master branch multiple times and retried the failed test multiple times but it always fails.

[marwan116]

Thanks for checking @MortalHappiness — it's unrelated to this PR.

The failing target is //python/ray/train:test_state, and the real error is an assertion rather than the raylet noise in the same stderr block:

>   assert dataset_info.dataset_uuid == dataset._uuid
E   AssertionError: assert '0342l3gti8UINtnldFbV9Y' == '0342l3WtNOxOM6uop7CGAJ'
python/ray/train/tests/test_state.py:297

It broke on master in #65075, which regenerates a dataset's uuid on deserialization, and was fixed on master by #65290 (merged 2026-08-07 17:28 UTC). Build #71537 ran at 01:43 UTC that day, ~16h before the fix landed — the failure was deterministic, which is why the retries and master syncs never cleared it.

I've merged current master (including f22677d) into the branch and pushed, so this target should be green on the next run.

For completeness: this PR touches only the dashboard reporter, the profiling defaults in ray_constants.py, and the frontend. test_state.py imports none of that, and there are no C++ or BUILD changes.