mcs/scheduling: prevent scheduler update notifier from blocking

[rleungx]

What problem does this PR solve?

Issue Number: Close #9776

What is changed and how does it work?

Make scheduler update notifications non-blocking so config updates only enqueue one pending signal when the scheduler updater is busy or the notifier channel is full.

Clear the scheduler update notifier registered by updateScheduler when the scheduling primary steps down, while preserving a newer notifier registered by a subsequent primary.

Check List

Tests

• Unit test

Related changes

• Need to cherry-pick to the release branch

Release note

Fix the issue that the scheduling service may stop working after switching primary while scheduler configs are updated.

Summary by CodeRabbit

• Bug Fixes

  • Notifications for scheduler updates are now non-blocking and coalesced to avoid hangs.
  • Notifier state is reliably cleared during shutdown or when replaced, improving cleanup.

• Tests

  • Added tests covering notifier lifecycle, replacement/clearing semantics, non-blocking behavior, and goroutine leak detection.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 46abadb0-9b37-40d6-b33c-90b3d352a94e

📥 Commits

Reviewing files that changed from the base of the PR and between 773fb37 and e164654.

📒 Files selected for processing (1)

• pkg/mcs/scheduling/server/config/config_test.go

🚧 Files skipped from review as they are similar to previous changes (1)

• pkg/mcs/scheduling/server/config/config_test.go

---

📝 Walkthrough

Walkthrough

Updated scheduler-updating notifier mechanism to prevent blocking during primary switches. Changed channel type to bidirectional, implemented non-blocking notification coalescing, added atomic clearing with compare-and-swap, ensured cleanup on exit, and validated with lifecycle tests.

Changes

Scheduler Notifier Lifecycle Management

Layer / File(s)	Summary
Notifier channel type and non-blocking API pkg/mcs/scheduling/server/config/config.go	PersistConfig notifier type changed to bidirectional chan struct{}. SetSchedulersUpdatingNotifier signature updated, ClearSchedulersUpdatingNotifier added with atomic compare-and-swap to conditionally clear. tryNotifySchedulersUpdating changed from unconditional blocking send to non-blocking coalesced send using select with default.
Cleanup on exit pkg/mcs/scheduling/server/cluster.go	updateScheduler adds deferred cleanup call to ClearSchedulersUpdatingNotifier on function exit, preventing notifier from remaining set after scheduler update listener stops.
Notifier lifecycle tests pkg/mcs/scheduling/server/config/config_test.go	Tests verify SetScheduleConfig does not block when notifier channel is full, notifier replacement and clearing semantics work correctly, and helper constructs test schedule configs.

Sequence Diagram(s)

sequenceDiagram
  participant UpdateScheduler as updateScheduler<br/>(cluster.go)
  participant PersistConfig as PersistConfig<br/>(config.go)
  participant Notifier as Notifier Channel

  UpdateScheduler->>PersistConfig: SetSchedulersUpdatingNotifier(ch)
  activate PersistConfig
  PersistConfig->>PersistConfig: Store ch in atomic.Value
  deactivate PersistConfig

  loop Background Loop
    UpdateScheduler->>PersistConfig: SetScheduleConfig(...)
    activate PersistConfig
    PersistConfig->>PersistConfig: tryNotifySchedulersUpdating()
    PersistConfig->>Notifier: select { case ch <- struct{}{} | default: }
    Note over Notifier: Non-blocking send<br/>Coalesces multiple signals
    deactivate PersistConfig
  end

  Note over UpdateScheduler: Function exit or shutdown
  UpdateScheduler->>PersistConfig: defer ClearSchedulersUpdatingNotifier(ch)
  activate PersistConfig
  PersistConfig->>PersistConfig: CompareAndSwap in atomic.Value<br/>Clear only if ch matches
  deactivate PersistConfig

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Poem

🐰 A notifier cleared with care,
Non-blocking sends float in air,
When primaries hop and trade,
No channel trap will be laid.
Atomic swap and deferred goodbye — hop by hop, we fly!

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	Title clearly and specifically describes the main change: preventing scheduler update notifier from blocking, which is the core issue addressed in this PR.
Description check	✅ Passed	Description includes issue number, detailed commit message explaining the changes, test coverage confirmation, and release note. Follows the required template structure.
Linked Issues check	✅ Passed	Code changes directly address issue #9776 by implementing non-blocking notifications and clearing notifiers during primary step-down to prevent blocking.
Out of Scope Changes check	✅ Passed	All changes focus on scheduler update notifier handling during primary transitions and are directly related to fixing issue #9776. No extraneous modifications detected.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[codecov]

Codecov Report

❌ Patch coverage is 86.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.09%. Comparing base (9f1c47b) to head (e164654).
⚠️ Report is 39 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10869      +/-   ##
==========================================
- Coverage   79.17%   79.09%   -0.08%     
==========================================
  Files         536      536              
  Lines       73882    74044     +162     
==========================================
+ Hits        58494    58568      +74     
- Misses      11271    11336      +65     
- Partials     4117     4140      +23     

Flag	Coverage Δ
unittests	79.09% <86.66%> (-0.08%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[rleungx]

/test pull-integration-realcluster-test

[bufferflies]

Do we need to close the current channel? And if the equal condition is not true, Dose it cause the channel leak(I don't find any where close it )?

[rleungx]

I do not think we should close the notifier here. updateScheduler exits through c.ctx.Done(), and closing the channel could race with tryNotifySchedulersUpdating after it has loaded the channel and cause a send-on-closed-channel panic. If current != notifier, it means a newer notifier has already been registered, so the old one should not clear or close it. The old channel will be released after the old updateScheduler exits.

[King-Dylan]

/label needs-cherry-pick-release-8.5

[ti-chi-bot]

[LGTM Timeline notifier]

Timeline:

• 2026-06-25 12:34:17.228058514 +0000 UTC m=+210140.891284027: ☑️ agreed by bufferflies.
• 2026-06-26 03:05:00.288754037 +0000 UTC m=+262383.951979550: ☑️ agreed by lhy1024.

[ti-chi-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bufferflies, lhy1024, niubell

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

• OWNERS [bufferflies,lhy1024]
• pkg/mcs/scheduling/server/config/OWNERS [niubell]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ti-chi-bot]

In response to a cherrypick label: new pull request created to branch release-8.5: #10955.