Skip to content

test: stabilize flaky resource manager client tests#11011

Open
rleungx wants to merge 1 commit into
tikv:masterfrom
rleungx:fix-resource-manager-client-flaky-tests
Open

test: stabilize flaky resource manager client tests#11011
rleungx wants to merge 1 commit into
tikv:masterfrom
rleungx:fix-resource-manager-client-flaky-tests

Conversation

@rleungx

@rleungx rleungx commented Jul 16, 2026

Copy link
Copy Markdown
Member

What problem does this PR solve?

Issue Number: Close #10650, Close #8512, Close #8739, Close #7400, Ref #6810

Several tests in TestResourceManagerClientTestSuite assume that asynchronous
resource-manager state is visible immediately or include unrelated wall-clock
overhead in timing assertions. These assumptions make the suite flaky under CI
load, especially with standalone resource-manager discovery.

What is changed and how does it work?

Stabilize resource-manager client integration tests without changing the tested
behavior.

Wait until standalone resource-manager metadata and RU statistics are observable
before checking their values. Measure token-bucket wait time using the duration
reported by OnRequestWait so scheduler and logging overhead do not affect the
throttling assertion.

The existing final assertions remain in place. The changes only replace
immediate reads and fixed sleeps with condition-based waits and use the
controller-reported wait duration for the timing check.

Check List

Tests

  • Integration test
    • Affected tests in both deployment modes, 3 repetitions: PASS (194.898s)
    • Complete TestResourceManagerClientTestSuite in both deployment modes: PASS (127.256s)
    • git diff --check: PASS

Release note

None.

Summary by CodeRabbit

  • Tests
    • Improved integration tests to reliably handle asynchronous resource-group updates and service failover.
    • Added eventual-state checks for resource visibility, statistics, fill rates, and leader changes.
    • Updated request timing assertions to use reported wait durations for more consistent validation.

Wait for standalone resource manager metadata and RU statistics to become observable before asserting their values. Measure token bucket wait time using the duration returned by the controller so host scheduling delays do not make timing assertions flaky.

Signed-off-by: Ryan Leung <rleungx@gmail.com>
@ti-chi-bot ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. dco-signoff: yes Indicates the PR's author has signed the dco. labels Jul 16, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign andremouche for approval. For more information see the Code Review Process.
Please ensure that each of them provides their approval before proceeding.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot ti-chi-bot Bot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Jul 16, 2026
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 490a4024-6507-4e56-8b4e-b2e4c77b16e0

📥 Commits

Reviewing files that changed from the base of the PR and between 6a2787b and 45bccd7.

📒 Files selected for processing (1)
  • tests/integrations/mcs/resourcemanager/resource_manager_test.go

📝 Walkthrough

Walkthrough

Integration tests now account for asynchronous resource-group propagation by polling for visibility, controller state, RU statistics, and fill-rate updates. Controller timing assertions use durations returned by request handling instead of measuring elapsed wall-clock time.

Changes

Resource manager integration test stability

Layer / File(s) Summary
Watch state synchronization
tests/integrations/mcs/resourcemanager/resource_manager_test.go
Watch tests wait for resource-group visibility, controller activation, and updated fill rates before asserting state.
Controller wait accounting
tests/integrations/mcs/resourcemanager/resource_manager_test.go
Request timing checks accumulate durations returned by OnRequestWait for read and write requests.
Consumption and failover assertions
tests/integrations/mcs/resourcemanager/resource_manager_test.go
RU-statistics and failover checks use eventual polling until resource-group objects and fill rates match expected values.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

  • tikv/pd#10873: Updates resource-manager integration tests for asynchronous resource-group loading and visibility.
  • tikv/pd#10935: Uses eventual resource-group retrieval before asserting updated state.

Suggested labels: lgtm, approved, ok-to-test

Suggested reviewers: bufferflies, lhy1024

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: stabilizing flaky resource manager client tests.
Description check ✅ Passed The PR description matches the template with problem statement, issue links, changes, tests, and release note.
Linked Issues check ✅ Passed The changes address the listed flaky tests by waiting for resource-group visibility and using reported wait durations, matching the linked issue goals.
Out of Scope Changes check ✅ Passed The diff stays within the test file and only adjusts flaky-test timing and eventual checks, with no unrelated code changes.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dco-signoff: yes Indicates the PR's author has signed the dco. release-note-none Denotes a PR that doesn't merit a release note. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.

Projects

None yet

1 participant