[release-nextgen-202603] client/resource_group: add client-side demand observability by YuhaoZhang00 · Pull Request #10866 · tikv/pd

YuhaoZhang00 · 2026-06-10T05:52:31Z

What problem does this PR solve?

Issue Number: ref #10488, ref #10581

Backport of #10604 to release-nextgen-202603, plus some other metrics.

Related dashboard PR: pingcap/tidb#69090

What is changed and how does it work?

This PR adds client-side Resource Control observability on the release-nextgen branch. The metrics are organized by the Resource Control diagnosis chain:

Client request cost accounting and consumption sampling

resource_manager_client_token_request_consume
- Histogram of RU cost per client accounting event.
- Used by the TiDB dashboard RU Cost Distribution panel.
resource_manager_client_resource_group_consume_by_type{type="rru|wru", direction="charge|refund"}
- Counter of client-side RU consumption flow split by read RU/write RU and charge/refund direction.
- Failed write payback is recorded as direction="refund" because Prometheus counters cannot decrease.
- The TiDB dashboard RU Consumption By Type panel shows net actual consumption as rate(charge) - rate(refund).
resource_manager_client_resource_read_byte_sum
resource_manager_client_resource_write_byte_sum{direction="charge|refund"}
resource_manager_client_resource_kv_cpu_time_ms_sum
resource_manager_client_resource_sql_cpu_time_ms_sum
- Component cost counters for read bytes, write bytes, KV CPU time, and SQL CPU time.
- Failed write byte payback is recorded as direction="refund" on resource_manager_client_resource_write_byte_sum.
- Used by the TiDB dashboard Client Resource Bytes and Client CPU Cost panels.

Client pre-throttling demand

resource_manager_client_resource_group_demand_ru_per_sec
- EMA of RU demand before entering the local limiter.
- Demand is sampled from the client consumption paths (OnRequestWait, OnResponse, OnResponseWait, and ReportConsumption) rather than only from acquireTokens, so burstable and response-side consumption paths are covered.
- Helps distinguish low demand from demand that is being throttled.
- Used by the TiDB dashboard RU Demand, Grant And Fill Rate panel.

Client limiter local token state

resource_manager_client_resource_group_token_balance
resource_manager_client_resource_group_fill_rate
resource_manager_client_resource_group_burst_limit
resource_manager_client_resource_group_avg_ru_per_sec
resource_manager_client_resource_group_throttled
- Exposes the local limiter state after PD token bucket responses are applied.
- Used by the TiDB dashboard RU Demand, Grant And Fill Rate and Token Balance And Throttle panels.

Client-side actual grant observation

resource_manager_client_resource_group_actual_grant_tokens
- Counter of raw RU tokens actually granted by PD in token bucket responses.
- Recorded before trickle carry-over is merged into the local limiter state, so it reflects the raw grant received by this client.
- Used by the TiDB dashboard RU Demand, Grant And Fill Rate panel as a 1-minute actual grant rate curve.

Client wait and token bucket request diagnosis

Reuses existing token request metrics:
- resource_manager_client_token_request_duration
- resource_manager_client_token_request_resource_group
- resource_manager_client_token_request_low_token_notified
Adds structured slow-path logs for degraded mode, slow token requests, long waits, and throttling.
Logs include fields such as client_unique_id, resource_group, available_tokens, fill_rate, avg_ru_per_sec, demand_ru_per_sec, throttled, and wait_duration.

Metric cleanup

Removes stale label values when resource groups are cleaned up, including the new client-side metrics and their charge/refund label values.

Validation

make gotest GOTEST_ARGS='./resource_group/controller/... -count=1' from client/
git diff --check

Notes

This PR provides the PD client metrics consumed by the appended Resource Control diagnosis panels in pingcap/tidb#69090.

This is intentionally draft. Additional release-nextgen-specific changes may be added before marking ready for review.

Release note

Add client-side resource control metrics for pre-throttling demand, actual granted tokens, limiter state, and per-instance RU consumption breakdowns.

Backport PR tikv#10604 to release-nextgen-202603. Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

coderabbitai · 2026-06-10T05:52:40Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 66d99872-3aa0-4a66-86b8-38113c1e2ed5

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

ti-chi-bot · 2026-06-10T05:52:42Z

Hi @YuhaoZhang00. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

rleungx · 2026-06-10T07:05:14Z

/ok-to-test

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

rleungx · 2026-06-10T10:04:37Z

 	logControllerTrace("[resource group controller] update run state", zap.String("name", gc.name), zap.Any("request-unit-consumption", gc.run.consumption), zap.Bool("is-throttled", gc.isThrottled.Load()))
 	gc.run.now = newTime
+
+	gc.observeComponentConsumption(&trickleDelta)


This records the SQL CPU metric from the zeroed consumption baseline, so the first sample in single-group-by-keyspace mode includes all process CPU since startup. Can we subtract the initial SqlLayerCpuTimeMs baseline before observing this metric?

rleungx · 2026-06-10T10:04:37Z

 	gc.mu.Lock()
 	add(gc.mu.consumption, consumption)
 	gc.mu.Unlock()
+	gc.observeComponentConsumption(consumption)


ReportConsumption can include RRU/WRU, for example TiFlash MPP cost, but this path only records component metrics. Please also call observeConsumption here so consume_by_type does not undercount direct reports.

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

rleungx · 2026-06-10T13:07:36Z

+	requestSubsystem       = "request"
+	tokenRequestSubsystem  = "token_request"
+	resourceGroupSubsystem = "resource_group"
+	resourceUnitSubsystem  = "resource_unit"


This constant is unused. Please remove it or wire it into a metric; otherwise staticcheck may fail with U1000.

rleungx · 2026-06-10T13:07:36Z

+			log.Warn("[resource group controller] waited for tokens", gc.logFields(waitDuration, nil)...)
+		}
 	}
+	gc.observeConsumption(delta)


This records write WRU/write bytes before the response is known. If the write later fails, the payback delta is negative but these counters cannot be decremented, so consume_by_type{type="wru"} and write_byte_sum can overcount failed writes. Can we observe actual charged consumption after the final response delta instead?

rleungx · 2026-06-10T13:07:36Z

 			counter.limiter.RemoveTokens(time.Now(), v)
 		}
 	}
+	gc.observeConsumption(delta)


This path observes response-side consumption, but demand_ru_per_sec is only updated inside acquireTokens. Callers using OnResponse will undercount read/CPU demand; can we sample demand here before removing tokens as well?

rleungx · 2026-06-10T13:07:37Z

+}
+
+func (gc *groupCostController) calcDemandAvg(counter *tokenCounter, delta float64) bool {
+	if delta <= 0 {


When traffic stops, this returns without updating the EMA, so the gauge can keep the last non-zero demand forever. Can the periodic state update refresh it with unchanged demandTotalRU so it decays toward zero?

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

rleungx · 2026-06-11T02:55:33Z

+	if v := getRUValueFromConsumption(delta); v > 0 {
+		gc.metrics.consumeTokenHistogram.Observe(v)
+	}
+	observeCounterDelta(gc.metrics.chargeTokenByTypeRRU, gc.metrics.refundTokenByTypeRRU, delta.RRU)


ReportConsumption can still pass a nil consumption here. The old add path ignored nil, but this now dereferences delta, so please guard observeConsumption with if delta == nil { return } like observeComponentConsumption.

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

YuhaoZhang00 · 2026-06-11T03:04:03Z

/cc JMPotato

rleungx · 2026-06-11T03:09:27Z

 	for _, calc := range gc.calculators {
 		calc.BeforeKVRequest(delta, info)
 	}
+	gc.observeDemand(delta)


This is on the per-request hot path before token acquisition. Since recordDemand takes avgMu for every request, can we verify the P99/P999 impact under a high-QPS single resource group, or consider sampling/aggregation if contention shows up?

Moved demand accumulation off avgMu: the request path now uses a scaled atomic Add, while the periodic updater loads the accumulated demand and updates the EMA under avgMu.

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

rleungx · 2026-06-11T04:11:40Z

+		return
+	}
+	if v := getRUValueFromConsumption(delta); v > 0 {
+		gc.metrics.consumeTokenHistogram.Observe(v)


This histogram is updated on the client consumption hot path. Since histograms are heavier than counters/gauges, can we confirm the P99/P999 latency impact under high QPS, or consider sampling/periodic aggregation if the overhead is visible?

Moved this histogram back under the original trace/debug gate (align with base behaviour). counters remain always collected

…ackport-10604-release-nextgen-202603

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

codecov · 2026-06-11T04:51:22Z

Codecov Report

❌ Patch coverage is 94.30380% with 18 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (release-nextgen-202603@0b6989b). Learn more about missing BASE report.

Additional details and impacted files

@@                    Coverage Diff                    @@
##             release-nextgen-202603   #10866   +/-   ##
=========================================================
  Coverage                          ?   78.99%           
=========================================================
  Files                             ?      531           
  Lines                             ?    71922           
  Branches                          ?        0           
=========================================================
  Hits                              ?    56815           
  Misses                            ?    11079           
  Partials                          ?     4028

Flag	Coverage Δ
unittests	`78.99% <94.30%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

YuhaoZhang00 · 2026-06-11T05:26:20Z

/retest

YuhaoZhang00 · 2026-06-11T05:40:44Z

/retest

ti-chi-bot · 2026-06-11T06:18:28Z

[LGTM Timeline notifier]

Timeline:

2026-06-11 05:18:09.487442299 +0000 UTC m=+1023590.557759688: ☑️ agreed by rleungx.
2026-06-11 06:18:27.654188266 +0000 UTC m=+1027208.724505666: ☑️ agreed by JmPotato.

ti-chi-bot · 2026-06-11T06:28:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: JmPotato, niubell, rleungx

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [JmPotato,rleungx]
~~client/resource_group/controller/OWNERS~~ [niubell]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

ti-chi-bot · 2026-06-11T06:46:44Z

@YuhaoZhang00: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-unit-test-next-gen-1	`3c2e60c`	link	unknown	`/test pull-unit-test-next-gen-1`
pull-unit-test-next-gen-2	`3c2e60c`	link	unknown	`/test pull-unit-test-next-gen-2`
pull-unit-test-next-gen-3	`3c2e60c`	link	unknown	`/test pull-unit-test-next-gen-3`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

client/resource_group: add client-side demand observability

6b45b01

Backport PR tikv#10604 to release-nextgen-202603. Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

ti-chi-bot Bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has signed the dco. labels Jun 10, 2026

ti-chi-bot Bot added contribution This PR is from a community contributor. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jun 10, 2026

ti-chi-bot Bot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Jun 10, 2026

ti-chi-bot Bot added ok-to-test Indicates a PR is ready to be tested. and removed needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jun 10, 2026

client/resource_group: add actual grant observability

2758f8f

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

YuhaoZhang00 marked this pull request as ready for review June 10, 2026 09:32

ti-chi-bot Bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jun 10, 2026

rleungx reviewed Jun 10, 2026

View reviewed changes

client/resource_group: fix consumption metric accounting

6b0ddbe

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

YuhaoZhang00 requested a review from rleungx June 10, 2026 10:52

rleungx reviewed Jun 10, 2026

View reviewed changes

YuhaoZhang00 added 3 commits June 11, 2026 08:59

client/resource_group: remove unused metric subsystem

4c531bc

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

client/resource_group: sample demand from consumption paths

9ad43bd

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

client/resource_group: record consumption refunds

7079136

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

rleungx reviewed Jun 11, 2026

View reviewed changes

YuhaoZhang00 added 2 commits June 11, 2026 10:56

client/resource_group: refresh demand metric periodically

1359015

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

client/resource_group: handle nil consumption metrics

42a4b4f

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

YuhaoZhang00 requested a review from rleungx June 11, 2026 03:03

ti-chi-bot Bot requested a review from JmPotato June 11, 2026 03:04

rleungx reviewed Jun 11, 2026

View reviewed changes

client/resource_group: reduce demand metric contention

3001854

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

YuhaoZhang00 requested a review from rleungx June 11, 2026 03:58

rleungx reviewed Jun 11, 2026

View reviewed changes

YuhaoZhang00 added 3 commits June 11, 2026 12:12

Merge remote-tracking branch 'upstream/release-nextgen-202603' into b…

cbbc039

…ackport-10604-release-nextgen-202603

client/resource_group: keep test helper compatible

1a17fad

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

client/resource_group: gate consume histogram on trace

584262f

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

YuhaoZhang00 requested a review from rleungx June 11, 2026 04:45

YuhaoZhang00 added 2 commits June 11, 2026 13:04

client/resource_group: avoid direct metric model test dependency

3c8b848

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

client/resource_group: name average metric results

3c2e60c

Signed-off-by: Yuhao Zhang <yhzhang00@outlook.com>

rleungx approved these changes Jun 11, 2026

View reviewed changes

ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jun 11, 2026

JmPotato approved these changes Jun 11, 2026

View reviewed changes

ti-chi-bot Bot added lgtm and removed needs-1-more-lgtm Indicates a PR needs 1 more LGTM. labels Jun 11, 2026

niubell approved these changes Jun 11, 2026

View reviewed changes

ti-chi-bot Bot added the approved label Jun 11, 2026

ti-chi-bot Bot merged commit 491a11c into tikv:release-nextgen-202603 Jun 11, 2026
43 of 48 checks passed

This was referenced Jul 13, 2026

[release-nextgen-202603] client: remove successful token wait logs #10997

Merged

client/resource_group: add client-side demand observability #10604

Open

Uh oh!

Conversation

YuhaoZhang00 commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

What is changed and how does it work?

Validation

Notes

Release note

Uh oh!

coderabbitai Bot commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

ti-chi-bot Bot commented Jun 10, 2026

Uh oh!

rleungx commented Jun 10, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

YuhaoZhang00 commented Jun 11, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Jun 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

YuhaoZhang00 commented Jun 11, 2026

Uh oh!

YuhaoZhang00 commented Jun 11, 2026

Uh oh!

ti-chi-bot Bot commented Jun 11, 2026

[LGTM Timeline notifier]

Uh oh!

ti-chi-bot Bot commented Jun 11, 2026

Uh oh!

Uh oh!

ti-chi-bot Bot commented Jun 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

YuhaoZhang00 commented Jun 10, 2026 •

edited

Loading

coderabbitai Bot commented Jun 10, 2026 •

edited

Loading

codecov Bot commented Jun 11, 2026 •

edited

Loading