Skip to content

test: stabilize flaky TestHotTestSuite/TestBuckets#10937

Open
flaky-claw wants to merge 1 commit into
tikv:masterfrom
flaky-claw:flakyfixer/case_cf396bbf1769-a6
Open

test: stabilize flaky TestHotTestSuite/TestBuckets#10937
flaky-claw wants to merge 1 commit into
tikv:masterfrom
flaky-claw:flakyfixer/case_cf396bbf1769-a6

Conversation

@flaky-claw

@flaky-claw flaky-claw commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #10651

Problem Summary:
Flaky test TestHotTestSuite/TestBuckets in github.com/tikv/pd/tools/pd-ctl/tests/hot intermittently fails, so this PR stabilizes that path.

What changed and how does it work?

Root Cause

TestBuckets teardown could call CleanCache while a late async write hot-cache task still touched HotPeerCache maps.

Fix

The final write drain serializes teardown behind the last write-side hot-cache task without changing CLI assertions or product behavior.

Verification

Spec:

  • target: github.com/tikv/pd/tools/pd-ctl/tests/hot :: TestHotTestSuite/TestBuckets
  • strategy: pd.issue_scoped.v1
  • plan mode: BASELINE_ONLY
  • requirements: required case must execute; no skip; repeat count = 1
  • baseline gates: required_flaky_gate, build_safety_gate, intent_guard_gate
  • feedback surface source: baseline_only

Observed result:

  • status: failed
  • required case executed: yes
  • submission decision: ALLOWED
  • note: Required flaky case executed during validation.
    Required flaky case was not skipped.
    nested_target_json passed.
    race_target passed.

Gate checklist:

  • raw_target_path_mismatch: SKIPPED
  • raw_package_path_mismatch: SKIPPED
  • nested_target_json: PASS
  • race_target: PASS
  • build: BLOCKED

Commands:

  • cd tools && go test -json -tags without_dashboard ./pd-ctl/tests/hot -run '^TestHotTestSuite/TestBuckets$' -count=1
  • cd tools && go test -race -tags without_dashboard ./pd-ctl/tests/hot -run '^TestHotTestSuite/TestBuckets$' -count=5

Release note

None

Fixes #10651

Summary by CodeRabbit

  • Tests
    • Added an extra assertion to verify hot peer statistics are present after asynchronous hot-peer updates, improving confidence that hot-spot handling behaves correctly.

@ti-chi-bot ti-chi-bot Bot added release-note-none Denotes a PR that doesn't merit a release note. contribution This PR is from a community contributor. dco-signoff: no Indicates the PR's author has not signed dco. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. labels Jun 24, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

Hi @flaky-claw. Thanks for your PR.

I'm waiting for a tikv member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ti-chi-bot ti-chi-bot Bot added the size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. label Jun 24, 2026
@coderabbitai

coderabbitai Bot commented Jun 24, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: fae75cf3-e879-4418-8c8b-6f2200b782b7

📥 Commits

Reviewing files that changed from the base of the PR and between 94277ce and 14674d3.

📒 Files selected for processing (1)
  • tools/pd-ctl/tests/hot/hot_test.go
🚧 Files skipped from review as they are similar to previous changes (1)
  • tools/pd-ctl/tests/hot/hot_test.go

📝 Walkthrough

Walkthrough

TestBuckets now verifies that write hot peer stats exist after asynchronous updates are drained and before teardown clears the hot cache.

Changes

Hot peer cache test

Layer / File(s) Summary
Bucket stats assertion
tools/pd-ctl/tests/hot/hot_test.go
Adds a non-nil check for hotStat.GetHotPeerStats(utils.Write, 0) after draining asynchronous hot peer updates.

Estimated code review effort: 1 (Trivial) | ~2 minutes

Possibly related PRs

  • tikv/pd#10863: Also updates checkBuckets to assert hot peer stats are populated after asynchronous updates.

Suggested labels: lgtm

Suggested reviewers: lhy1024, rleungx

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly matches the main change: stabilizing the flaky TestHotTestSuite/TestBuckets test.
Description check ✅ Passed The description includes the issue number, problem summary, fix summary, verification, and release note status.
Linked Issues check ✅ Passed The change addresses the reported HotPeerCache race by draining async writes before teardown, matching issue #10651.
Out of Scope Changes check ✅ Passed The only added assertion stays within the targeted test stabilization and does not introduce unrelated scope.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@tools/pd-ctl/tests/hot/hot_test.go`:
- Line 558: The drained write stats check in hot_test.go is vacuous because
hotStat.GetHotPeerStats(utils.Write, 0) returns an allocated map even when
empty, so reusing re.NotNil does not verify population. Update the assertion
near the existing hotStat.GetHotPeerStats(utils.Write, 0) call in the hot test
to check for a non-empty result instead, using a size/length-based assertion or
equivalent on the returned map so the test actually confirms drained write peers
were recorded.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e871b664-3b07-479c-bd56-d6d5a427c97a

📥 Commits

Reviewing files that changed from the base of the PR and between 51f4146 and 94277ce.

📒 Files selected for processing (1)
  • tools/pd-ctl/tests/hot/hot_test.go

// Drain async hot peer updates before TearDownTest resets the hot cache directly.
re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))
re.NotNil(hotStat.GetHotPeerStats(utils.Read, 0))
re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Use a non-vacuous assertion for drained write stats

Line 558 duplicates Line 556, and NotNil is ineffective here because GetHotPeerStats returns an allocated map even when no hot peers exist. This does not validate the drain/population condition.

Suggested fix
-	re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))
-	re.NotNil(hotStat.GetHotPeerStats(utils.Read, 0))
-	re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))
+	writeStats := hotStat.GetHotPeerStats(utils.Write, 0)
+	re.NotEmpty(writeStats)
+	found := false
+	for _, peers := range writeStats {
+		if len(peers) > 0 {
+			found = true
+			break
+		}
+	}
+	re.True(found)
+	re.NotNil(hotStat.GetHotPeerStats(utils.Read, 0))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
re.NotNil(hotStat.GetHotPeerStats(utils.Write, 0))
writeStats := hotStat.GetHotPeerStats(utils.Write, 0)
re.NotEmpty(writeStats)
found := false
for _, peers := range writeStats {
if len(peers) > 0 {
found = true
break
}
}
re.True(found)
re.NotNil(hotStat.GetHotPeerStats(utils.Read, 0))
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@tools/pd-ctl/tests/hot/hot_test.go` at line 558, The drained write stats
check in hot_test.go is vacuous because hotStat.GetHotPeerStats(utils.Write, 0)
returns an allocated map even when empty, so reusing re.NotNil does not verify
population. Update the assertion near the existing
hotStat.GetHotPeerStats(utils.Write, 0) call in the hot test to check for a
non-empty result instead, using a size/length-based assertion or equivalent on
the returned map so the test actually confirms drained write peers were
recorded.

@ti-chi-bot ti-chi-bot Bot added the needs-1-more-lgtm Indicates a PR needs 1 more LGTM. label Jul 8, 2026
@ti-chi-bot

ti-chi-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: lhy1024

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@ti-chi-bot

ti-chi-bot Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

[LGTM Timeline notifier]

Timeline:

  • 2026-07-08 11:39:56.005970371 +0000 UTC m=+195382.042065426: ☑️ agreed by lhy1024.

@ti-chi-bot ti-chi-bot Bot added the approved label Jul 8, 2026
Signed-off-by: Ryan Leung <rleungx@gmail.com>
@rleungx rleungx force-pushed the flakyfixer/case_cf396bbf1769-a6 branch from 94277ce to 14674d3 Compare July 16, 2026 02:44
@ti-chi-bot ti-chi-bot Bot added dco-signoff: yes Indicates the PR's author has signed the dco. and removed dco-signoff: no Indicates the PR's author has not signed dco. labels Jul 16, 2026
@rleungx

rleungx commented Jul 16, 2026

Copy link
Copy Markdown
Member

/retest

@ti-chi-bot

ti-chi-bot Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

@flaky-claw: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-unit-test-next-gen-3 14674d3 link true /test pull-unit-test-next-gen-3

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved contribution This PR is from a community contributor. dco-signoff: yes Indicates the PR's author has signed the dco. needs-1-more-lgtm Indicates a PR needs 1 more LGTM. needs-ok-to-test Indicates a PR created by contributors and need ORG member send '/ok-to-test' to start testing. release-note-none Denotes a PR that doesn't merit a release note. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

TestBuckets has a HotPeerCache data race

3 participants