Inference: Move tokenizer to a separate thread and scale frontend replicas with DP-size - #6223
Open
sidsingh-nvidia wants to merge 7 commits into
Open
Conversation
svcnvidia-nemo-ci
marked this pull request as draft
August 3, 2026 20:43
Contributor
|
This PR has been automatically converted to draft because all PRs must start as drafts. When you are ready for review, click Ready for Review to begin the review process. This will:
See the contribution guide for more details. |
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
wdykas
reviewed
Aug 5, 2026
sidsingh-nvidia
marked this pull request as ready for review
August 6, 2026 20:38
Contributor
Author
|
/ok to test 87dfbd8 |
wdykas
approved these changes
Aug 6, 2026
kvareddy
approved these changes
Aug 10, 2026
shanmugamr1992
approved these changes
Aug 11, 2026
svcnvidia-nemo-ci
added
Approved
Contributor
NeMo-RL functional testscluster: Exactly what was tested
Megatron-Bridge was pinned to Posted by the nemo-rl-testing-agent. Re-runs edit this comment in place. |
This was referenced Aug 19, 2026
Draft
Contributor
Author
|
/ok to test 929bf14 |
The replicas shared one listening socket: the parent bound it and passed the same fd to every forked worker, so all of them accepted from a single queue. That does not balance. Whichever worker is already running tends to win the wakeup, and it keeps winning, because an event loop with work in flight polls more often than one blocked in accept. SO_REUSEPORT was set on that socket but was inert -- the kernel only load-balances when several sockets are bound to the port and it can hash a connection's 4-tuple to choose between them. Measured with 32 replicas and a fresh connection per request, ~90% of traffic landed on 5 of them, 20 replicas served exactly one request each, and throughput was 3.7x lower than the same server under a pooled client that opened its connections up front. Load made it worse rather than averaging it out: at 2048 requests the busiest replica took 604x the quietest. Each replica now binds its own socket on the shared port, so every one gets its own accept queue. Spread became max/min 1.5x with all replicas serving, and throughput 2.5x on the fresh-connection path. A pooled client is unaffected in steady state, which is the point: how well the frontend spreads no longer depends on connection behaviour the server cannot observe. start_text_gen_server now returns the base URL it is serving on. Callers that start a frontend on more than one rank need the addresses to spread requests over; previously they had to reconstruct them. The signature is otherwise unchanged, including sock, which still fixes the port -- it is closed rather than shared, since replicas bind their own. tools/run_dynamic_text_generation_server.py gains --frontend-on-all-ranks, which hosts a frontend on every rank and gathers the URLs. Frontend work is CPU-bound and otherwise confined to one rank's CPU allocation while the rest of the job's cores go unused. Signed-off-by: Siddharth Singh <sidsingh@nvidia.com> (cherry picked from commit 1001ca9)
sidsingh-nvidia
force-pushed
the
siddharth/frontend-tokenize-thread
branch
from
August 19, 2026 22:23
929bf14 to
54522c4
Compare
Contributor
Author
|
/ok to test 890d9f2 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Issue tracking
For PRs from open-source community contributors:
Linked issue:
Contribution process
Pre-checks
Code review
Feel free to message or comment @NVIDIA/mcore-oncall to help accelerate your merge into main. The less complex your PR is, the faster it will be approved and merged!
All PRs start as draft. If you open a non-draft PR, it will be automatically converted to draft.
Step 1: Mark PR as "Ready for Review"
.github/CODEOWNERS.Final Review might get declined if these requirements are not fulfilled.
Step 2: Final Review
For PRs that change
megatron/core, once all expert reviewers have approved, theFinal Reviewlabel is applied automatically and final reviewers are assigned.For PRs outside
megatron/core, this step is skipped.Step 3: Approved
Once all required reviewers have approved, the
Approvedlabel is applied automatically.Merge
Any member of mcore-engineers will be able to merge your PR.