pd: report hot read cpu in heartbeat (#10178)

[lhy1024]

What problem does this PR solve?

Issue Number: ref #5718

This is a cherry-pick of #10178 to release-nextgen-202603.

What is changed and how does it work?

Report hot read CPU from store heartbeats and use it as a hot read scheduling dimension when the cluster version supports it.

Expose CPU read rates in hot-region API responses, hot-region history, store load statistics, pd-ctl output, and Grafana.

Keep compatibility by falling back to query/byte based priorities when CPU hot scheduling is not supported.

Cherry-pick source: #10178
Base branch: release-nextgen-202603
Cherry-picked commit: 3aee2efeb594346498f8a1eadd044ce212c2001a

Conflicts:

None

Check List

Tests

• Unit test
• Integration test

Code changes

• Has the configuration change
• Has HTTP APIs changed (Don't forget to add the declarative for the new API)

Related changes

• Need to cherry-pick to the release branch

Verification:

PATH="$PWD/.tools/bin:$PATH" make check
PATH="$PWD/.tools/bin:$PATH" make gotest GOTEST_ARGS='./pkg/statistics ./pkg/schedule/schedulers ./pkg/versioninfo -count=1'
(cd client && PATH="$PWD/../.tools/bin:$PATH" make gotest GOTEST_ARGS='./http -count=1')

Release note

Report hot read CPU in store heartbeats and expose CPU-based hot-read scheduling statistics.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b3f3e268-6941-4527-ae2f-2d85799f658f

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

❌ Patch coverage is 86.05769% with 29 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.17%. Comparing base (b7828f1) to head (76c5be0).
⚠️ Report is 5 commits behind head on release-nextgen-202603.

Additional details and impacted files

@@                    Coverage Diff                     @@
##           release-nextgen-202603   #10737      +/-   ##
==========================================================
+ Coverage                   79.07%   79.17%   +0.10%     
==========================================================
  Files                         531      532       +1     
  Lines                       72298    72818     +520     
==========================================================
+ Hits                        57168    57654     +486     
- Misses                      11093    11125      +32     
- Partials                     4037     4039       +2     

Flag	Coverage Δ
unittests	79.17% <86.05%> (+0.10%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[lhy1024]

Temporary note for the current TiDBX/CSE validation:

The test cluster is running a NextGen/CSE TiKV build from tidbcloud/cloud-storage-engine#5241:

• TiKV/CSE hash: 63aace1e17617f6b91edfba9e3660b41788b3407
• live TiKV version reported to PD: 8.5.4+branch-HEAD

That CSE PR already reports hot-read CPU stats in store heartbeat, but this PD PR originally gated CPU-based hot-read scheduling by cluster version only:

• >= 8.5.7, or
• >= 9.0.0-beta.1

Because the CSE test build reports itself as 8.5.4+branch-HEAD, PD falls back from the new default read-priorities = [cpu, byte] to the compatibility read-priorities = [query, byte]. Setting read-priorities manually is not enough, because the runtime compatibility adjustment still forces the fallback when cpuSupport=false.

I pushed d37d4add1 as a temporary NextGen-only bypass: if PD is built/running as NextGen, IsHotScheduleWithCPUSupported returns true regardless of the TiKV semantic version. This lets the current CSE validation actually exercise CPU-first hot-read scheduling.

This is intentionally scoped to NextGen and keeps the classic compatibility behavior unchanged. We should still discuss the long-term rule: either version CSE CPU reporting explicitly, use a capability signal instead of semantic version, or only keep this shortcut for NextGen release branches where the paired CSE build is controlled.

Validated locally:

go test ./pkg/versioninfo -run 'TestIsHotScheduleWithCPUSupported' -count=1
go test ./pkg/schedule/schedulers -run 'TestAdjustPrioritiesConfigCPUFallback' -count=1

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bufferflies, rleungx
Once this PR has been reviewed and has the lgtm label, please assign benmeadowcroft for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS [bufferflies,rleungx]
• pkg/schedule/schedulers/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-05-29 06:39:34.877390962 +0000 UTC m=+1315.252200647: ☑️ agreed by rleungx.
• 2026-06-10 11:04:47.252589158 +0000 UTC m=+957988.322906538: ☑️ agreed by bufferflies.

[lhy1024]

30-minute Benchbot validation for TiDBX looks normal overall.

Images under test:

• PD: pd: report hot read cpu in heartbeat (#10178) #10737
• TiKV / CSE: https://github.com/tidbcloud/cloud-storage-engine/pull/5241

Since Benchbot does not support paired TiDBX comparison tests yet, each workload was run independently for 3 rounds. The table below uses the average result.

Workload	Case	Throughput (cloud -> Hot)	Latency (cloud -> Hot)	Result
TPC-C	tpmC	tpmC: 118,910.7 -> 133,934.0 (+12.6%)	NEW_ORDER avg: 52.6 ms -> 47.0 ms (-10.6%)	✅ Better
TPC-C	tpmTotal	tpmTotal: 264,268.6 -> 297,621.1 (+12.6%)	NEW_ORDER avg: 52.6 ms -> 47.0 ms (-10.6%)	✅ Better
TPC-C	NEW_ORDER avg latency	tpmC: 118,910.7 -> 133,934.0 (+12.6%)	52.6 ms -> 47.0 ms (-10.6%)	✅ Better
TPC-C	NEW_ORDER p99 latency	tpmC: 118,910.7 -> 133,934.0 (+12.6%)	141.2 ms -> 88.1 ms (-37.6%)	✅ Better
TPC-C	NEW_ORDER p99.9 latency	tpmC: 118,910.7 -> 133,934.0 (+12.6%)	229.3 ms -> 118.8 ms (-48.2%)	✅ Better
Sysbench	oltp_point_select	47,169.9 -> 47,601.9 (+0.9%)	0.791 ms -> 0.763 ms (-3.5%)	➖ Flat
Sysbench	oltp_insert	12,126.1 -> 12,687.8 (+4.6%)	6.904 ms -> 6.668 ms (-3.4%)	✅ Better
Sysbench	oltp_update_non_index	12,200.0 -> 12,663.7 (+3.8%)	6.820 ms -> 6.614 ms (-3.0%)	✅ Better
Sysbench	oltp_update_index	10,853.3 -> 10,844.0 (-0.1%)	7.893 ms -> 7.939 ms (+0.6%)	➖ Flat
Sysbench	oltp_read_write	45,420.5 -> 44,126.3 (-2.8%)	1.135 ms -> 1.042 ms (-8.2%)	⚠️ Mixed
YCSB	workloada	12,340.8 -> 12,271.8 (-0.6%)	7.432 ms -> 7.495 ms (+0.8%)	➖ Flat
YCSB	workloadf	13,878.6 -> 14,026.8 (+1.1%)	6.498 ms -> 6.468 ms (-0.5%)	➖ Flat

Legend: ✅ Better = clear improvement, ➖ Flat = basically comparable, ⚠️ Mixed = one metric is lower while another improves.

Conclusion: the 30-minute validation does not show a significant regression. TPC-C improves clearly, most Sysbench cases are flat or better, and YCSB is also basically flat.

[lhy1024]

https://github.com/tidbcloud/cloud-storage-engine/pull/5241 has been merged

[lhy1024]

/retest

[lhy1024]

/retest

[lhy1024]

Updated the rollout strategy in commit 075454f93.

We will ship the hot-read CPU reporting, statistics, API, history, and Grafana visibility with this PR, but keep CPU-based scheduling as an explicit operational opt-in during the canary phase.

Behavior after this change:

• New clusters default to read-priorities = [query, byte].
• The NextGen CPU capability bypass remains in place, so an Ops override to [cpu, byte] is honored even when the paired CSE build reports an older semantic version.
• Classic clusters keep the existing version compatibility fallback.
• Invalid or legacy priority configuration falls back to the safe [query, byte] default when query stats are supported.

Enable CPU-first hot-read scheduling on a canary cluster:

pd-ctl -u "$PD_ADDR" scheduler config balance-hot-region-scheduler set read-priorities cpu,byte

Rollback:

pd-ctl -u "$PD_ADDR" scheduler config balance-hot-region-scheduler set read-priorities query,byte

This separates capability availability from scheduling rollout: Ops can enable selected canary clusters first, observe scheduling stability and performance, then expand coverage before we consider changing the default.

Validation completed:

make gotest GOTEST_ARGS="./pkg/schedule/schedulers -count=1"
NEXT_GEN=1 make gotest GOTEST_ARGS="./pkg/schedule/schedulers -count=1"

Both test modes passed. The tests now cover the safe default, explicit CPU opt-in, invalid-config fallback, and the NextGen capability state.

[lhy1024]

/retest

[lhy1024]

/retest

[lhy1024]

/retest

[lhy1024]

/retest

[lhy1024]

/retest

[lhy1024]

/retest

[lhy1024]

/retest