[autoscaler] Precompute SerializeToString keys to eliminate redundant serialization in scheduling loop

[Jade07-1]

Why are these changes needed?

After merging #64175, benchmark testing showed that scaling to 3000 nodes (15000 tasks × 0.2 CPU + 30MB memory) still takes ~51 minutes. Profiling with py-spy revealed two independent hotspots in the scheduling loop:

1. Pre-filter logic flaw — fixed in [autoscaler] Improve V2 scheduler pre-filter precision for multi-resource demands #65171 (OR-across-dimensions → AND-within-shape/OR-across-shapes)
2. Redundant SerializeToString calls — fixed in this PR

The UnschedulableRequestCache.contains() serializes each request via SerializeToString(deterministic=True) once per candidate node inside try_schedule, resulting in N_nodes × M_requests redundant serializations per scheduling round. For 15000 requests across 3000 nodes, this produces ~45M repeated protobuf serializations.

Profiling evidence (py-spy, 30s sample during baseline run):

• contains() in scheduler.py: 55% Own CPU (7.77s / 14s)
• _audit_fork_safety triggered by SerializeToString inside contains(): 18% Own CPU (2.43s / 14s)
• Combined: 73% of scheduler CPU spent on redundant serialization

Fix

Precompute a {id(request): bytes} dictionary once before entering the scheduling loop, and pass it through _sched_best_node → try_schedule. The UnschedulableRequestCache interface is simplified to accept raw bytes keys directly.

Includes a defensive .get() fallback: if a request is not in the precomputed dict, it falls back to on-the-fly serialization rather than raising a KeyError.

Benchmark

Cluster: 3000 max workers (1 CPU each), 15000 tasks × 0.2 CPU.

Condition	Time to 3000 nodes
After #64175 (baseline)	~51 min
+ #65171 only (pre-filter AND/OR)	~18 min
+ #65171 + this fix combined	~14 min

Related PRs

• [autoscaler] Improve V2 scheduler performance for large clusters  #64175 — baseline optimization that this PR builds on (benchmark results from that PR revealed this hotspot)
• [autoscaler] Improve V2 scheduler pre-filter precision for multi-resource demands #65171 — parallel fix for pre-filter logic (independent, can merge in either order)

Checks

• I've signed all my commits with DCO sign-off.
• I've run the existing tests — all pass. The try_schedule interface change is backward-compatible via Optional parameter with on-the-fly fallback.
• Changes are confined to python/ray/autoscaler/v2/scheduler.py (single file, low blast radius).
• No duplicate PR exists for this optimization (searched open PRs for SerializeToString, precompute, UnschedulableRequestCache — zero matches).
• AI assistance was used. I (human submitter) have reviewed every changed line and understand the change end-to-end.

[gemini-code-assist]

Code Review

This pull request optimizes the scheduling logic in python/ray/autoscaler/v2/scheduler.py by precomputing serialization keys (shape_keys) for resource requests, which avoids redundant SerializeToString calls inside the per-node scheduling loop. Additionally, UnschedulableRequestCache is simplified to cache and check these serialized shape keys directly. The feedback suggests simplifying the logic for retrieving or computing the serialization key sk inside try_schedule to eliminate duplicate calls to SerializeToString and improve readability.

[gemini-code-assist]

The logic for retrieving or computing the serialization key sk can be simplified to avoid duplicating the r.SerializeToString(deterministic=True) call. This improves readability and maintainability of the code.

            sk = shape_keys.get(id(r)) if shape_keys is not None else None
            if sk is None:
                sk = r.SerializeToString(deterministic=True)