feat(RL): add nvext Tokens-in-Tokens-Out and RL related response protocol and frontend support by biswapanda · Pull Request #9649 · ai-dynamo/dynamo

biswapanda · 2026-05-16T00:03:13Z

Summary

add Rust lib/llm protocol, preprocessor, and response plumbing for nvext Tokens-in-Tokens-Out
carry nvext.token_data, cache_salt, extra_fields, token constraints, prompt_logprobs, detokenize, and stop token IDs through the Rust request/response path
validate passthrough cache_salt and stop_token_ids; reject truncate_prompt_tokens until it is supported
add Rust unit coverage for chat/completion request extraction and nvext response fields

Scope

This PR is the Rust lib/llm half of the nvext Tokens-in-Tokens-Out split. It intentionally excludes the vLLM worker implementation so reviewers can focus on the public/internal request protocol and response mapping first.

Where To Review

lib/llm/src/protocols/openai/nvext.rs
lib/llm/src/preprocessor.rs
lib/llm/src/protocols/openai/completions.rs
lib/llm/src/protocols/openai/chat_completions.rs
lib/llm/src/protocols/openai/validate.rs

Local Tests

cargo test -p dynamo-llm protocols::openai::completions::tests:: --lib
cargo test -p dynamo-llm protocols::openai::chat_completions::tests:: --lib
rustfmt --edition 2024 --check lib/llm/src/protocols/openai/validate.rs lib/llm/src/protocols/openai/completions.rs lib/llm/src/protocols/openai/chat_completions.rs
git diff --check

Follow-up

The vLLM worker-side implementation should be reviewed separately so its diff remains scoped to components/src/dynamo/vllm/*.

Notes

supersedes #9382

Summary by CodeRabbit

New Features
- Added prompt log-probability data to API responses, enabling detailed token probability analysis at the prompt level
- Introduced new generation control options: detokenization toggle, allowed token IDs filtering, and bad word token IDs blocking for finer-grained output control
- Added completion token ID tracking in response payloads for improved token-level visibility

coderabbitai · 2026-05-16T00:08:26Z

Walkthrough

This PR extends the LLM backend and OpenAI protocol to surface prompt log-probabilities and additional sampling controls through request parsing, validation, preprocessing, response building, and delta emission across both chat and completion endpoints.

Changes

Prompt Log-Probabilities and Extended Sampling Support

Layer / File(s)	Summary
Core protocol types for prompt logprobs and sampling `lib/llm/src/protocols/common/llm_backend.rs`, `lib/llm/src/protocols/common.rs`	Introduces `PromptLogprobEntry` and `PromptLogprobs` types; adds optional `prompt_logprobs` field to `BackendOutput` and `LLMEngineOutput` with serde skip-serialization; extends `SamplingOptions` with `detokenize`, `allowed_token_ids`, and `bad_words_token_ids` fields.
OpenAI request protocol - CommonExt and trait implementations `lib/llm/src/protocols/openai/common_ext.rs`, `lib/llm/src/protocols/openai/chat_completions.rs`, `lib/llm/src/protocols/openai/completions.rs`, `lib/llm/src/protocols/openai.rs`	Extends `CommonExt` with new optional fields and implements `CommonExtProvider` trait methods on request types; implements `NvExtProvider` to expose unsupported_fields; updates `get_stop_token_ids()` to parse passthrough `stop_token_ids` from unsupported_fields; changes `get_prompt_logprobs()` to read from CommonExt or infer from echo setting; wires sampling options extraction into request processing.
Request validation for passthrough fields `lib/llm/src/protocols/openai/validate.rs`	Introduces `PASSTHROUGH_EXTRA_FIELDS` allowlist and updates validation to permit listed fields while type-checking `cache_salt` and `stop_token_ids`.
NvExt response protocol extensions `lib/llm/src/protocols/openai/nvext.rs`	Adds alternate DP rank header support; extends `NvExtProvider` trait and `NvExtResponse` with `completion_token_ids` and `prompt_logprobs` fields; extends response field selection with gating flags; updates `build_response_nvext` signature to accept backend inputs and conditionally populate fields based on selection and finish-reason; modifies aggregation to concatenate completion token IDs across chunks and overwrite prompt logprobs; adds `cache_salt` field to `NvExt`.
Request preprocessing and backend wiring `lib/llm/src/preprocessor.rs`, `lib/llm/src/backend.rs`, `lib/llm/src/protocols/unified.rs`	Adds `nvext_passthrough_args` helper for building nvext JSON; integrates into preprocessor request building and multimodal handling; populates max_tokens from remaining context length; wires prompt_logprobs from engine output into BackendOutput; adds CommonExtProvider delegation methods to UnifiedRequest.
Response emission through delta `lib/llm/src/protocols/openai/chat_completions/delta.rs`, `lib/llm/src/protocols/openai/completions/delta.rs`	Updates `choice_from_postprocessor` to capture and pass `prompt_logprobs` and `completion_token_ids` into `build_response_nvext`; updates debug logging to reference completion_token_ids field.
Test coverage `lib/llm/src/protocols/openai/chat_completions.rs`, `lib/llm/src/protocols/openai/completions.rs`, `lib/llm/src/protocols/openai/nvext.rs`, `lib/llm/src/migration.rs`, `lib/llm/tests/test_streaming_usage.rs`, `lib/llm/tests/tool_choice.rs`, `lib/llm/tests/tool_choice_finish_reasons.rs`	Adds tests for stop token ID passthrough, sampling token constraints, and truncate_prompt_tokens rejection; adds tito parity tests for completion_token_ids pass-through and prompt_logprobs final-chunk gating; updates all test fixtures to include `prompt_logprobs: None` in BackendOutput constructions.

🎯 4 (Complex) | ⏱️ ~45 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title mentions nvext tokens-in-tokens-out protocol and RL support, which directly aligns with the extensive changes across the LLM backend, protocol implementations, and response handling layers shown in the changeset.
Description check	✅ Passed	The pull request description is well-structured and complete, following the template with all required sections properly filled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

lib/llm/src/protocols/unified.rs (1)

279-287: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Delegate unsupported_fields() for UnifiedRequest.

OpenAIPreprocessor::nvext_passthrough_args relies on NvExtProvider::unsupported_fields() fallback (e.g., cache_salt). With the current impl, UnifiedRequest always uses the trait default None, so those fields are silently dropped on this path.

Suggested fix

 impl NvExtProvider for UnifiedRequest {
     fn nvext(&self) -> Option<&NvExt> {
         self.inner.nvext.as_ref()
     }

     fn raw_prompt(&self) -> Option<String> {
         None
     }
+
+    fn unsupported_fields(
+        &self,
+    ) -> Option<&std::collections::HashMap<String, serde_json::Value>> {
+        NvExtProvider::unsupported_fields(&self.inner)
+    }
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/llm/src/protocols/unified.rs` around lines 279 - 287, The UnifiedRequest
NvExtProvider impl currently only implements nvext() and raw_prompt(), so
unsupported_fields() falls back to the trait default and drops passthrough
fields; add an unsupported_fields(&self) implementation that delegates through
the inner nvext (e.g., return self.inner.nvext.as_ref().and_then(|nv|
nv.unsupported_fields())) so OpenAIPreprocessor::nvext_passthrough_args sees
fields like cache_salt; keep the implementation inside the impl NvExtProvider
for UnifiedRequest and reference the existing nvext() and raw_prompt() methods
for placement.

🧹 Nitpick comments (1)

lib/llm/src/protocols/openai/nvext.rs (1)
45-49: ⚡ Quick win

Add explicit test coverage for the DP-rank header alias path.

Line 48 introduces x-data-parallel-rank fallback, but there’s no direct test for alias-only input (or both headers present). Add a focused test to prevent regressions in routing behavior.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@lib/llm/src/protocols/openai/nvext.rs` around lines 45 - 49, Add unit tests
for the DP-rank header alias path to ensure routing uses HEADER_DP_RANK_ALIAS
when HEADER_DP_RANK is absent and that precedence is correct when both are
present: create tests that call the header-parsing logic in nvext.rs (exercising
the dp_rank extraction that uses HEADER_DP_RANK .or_else(HEADER_DP_RANK_ALIAS)
and the dp_rank variable) with (a) only HEADER_DP_RANK_ALIAS set and (b) both
HEADER_DP_RANK and HEADER_DP_RANK_ALIAS set, asserting the returned dp_rank
matches the expected value and that no panics or unexpected routing occur.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@lib/llm/src/protocols/unified.rs`:
- Around line 279-287: The UnifiedRequest NvExtProvider impl currently only
implements nvext() and raw_prompt(), so unsupported_fields() falls back to the
trait default and drops passthrough fields; add an unsupported_fields(&self)
implementation that delegates through the inner nvext (e.g., return
self.inner.nvext.as_ref().and_then(|nv| nv.unsupported_fields())) so
OpenAIPreprocessor::nvext_passthrough_args sees fields like cache_salt; keep the
implementation inside the impl NvExtProvider for UnifiedRequest and reference
the existing nvext() and raw_prompt() methods for placement.

---

Nitpick comments:
In `@lib/llm/src/protocols/openai/nvext.rs`:
- Around line 45-49: Add unit tests for the DP-rank header alias path to ensure
routing uses HEADER_DP_RANK_ALIAS when HEADER_DP_RANK is absent and that
precedence is correct when both are present: create tests that call the
header-parsing logic in nvext.rs (exercising the dp_rank extraction that uses
HEADER_DP_RANK .or_else(HEADER_DP_RANK_ALIAS) and the dp_rank variable) with (a)
only HEADER_DP_RANK_ALIAS set and (b) both HEADER_DP_RANK and
HEADER_DP_RANK_ALIAS set, asserting the returned dp_rank matches the expected
value and that no panics or unexpected routing occur.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: a6eb0b0e-a231-4196-8b4e-f258b6705f6f

📥 Commits

Reviewing files that changed from the base of the PR and between 7eb03b9 and a711882.

📒 Files selected for processing (17)

lib/llm/src/backend.rs
lib/llm/src/migration.rs
lib/llm/src/preprocessor.rs
lib/llm/src/protocols/common.rs
lib/llm/src/protocols/common/llm_backend.rs
lib/llm/src/protocols/openai.rs
lib/llm/src/protocols/openai/chat_completions.rs
lib/llm/src/protocols/openai/chat_completions/delta.rs
lib/llm/src/protocols/openai/common_ext.rs
lib/llm/src/protocols/openai/completions.rs
lib/llm/src/protocols/openai/completions/delta.rs
lib/llm/src/protocols/openai/nvext.rs
lib/llm/src/protocols/openai/validate.rs
lib/llm/src/protocols/unified.rs
lib/llm/tests/test_streaming_usage.rs
lib/llm/tests/tool_choice.rs
lib/llm/tests/tool_choice_finish_reasons.rs

biswapanda · 2026-05-16T00:28:14Z

@coderabbitai review

coderabbitai · 2026-05-16T00:28:20Z

✅ Actions performed

Review triggered.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

richardhuo-nv

LGTM!

Signed-off-by: Biswa Panda <biswa.panda@gmail.com>

biswapanda · 2026-05-22T23:56:50Z

Rebased on top of main.

Add `return_token_ids: Option<bool>` to `NvCreateChatCompletionRequest` and `NvCreateCompletionRequest`, a convenience alias for `nvext.extra_fields = ["completion_token_ids"]`. ## Why vLLM's OpenAI extension of the same name (`return_token_ids: true` in `extra_body`) is what clients like the `speculators` `data_generation_offline.py` capture flow and Prime-RL send today. A request that works against a raw vLLM server currently fails against a Dynamo-fronted vLLM server: ``` Validation: Unsupported parameter(s): `return_token_ids` ``` because the field lands in `unsupported_fields` (catch-all) and `validate_no_unsupported_fields` rejects it. This PR closes that gap without changing the response-side path — the existing nvext machinery already emits `completion_token_ids` on the response. ## What changes - Two new `Option<bool>` fields, one per request struct, with `#[serde(default, skip_serializing_if = "Option::is_none")]`. - Two `normalize_return_token_ids(&mut self)` methods that fold `Some(true)` into `nvext.extra_fields.push("completion_token_ids")` (idempotent — no duplicate entry if the caller also set the nvext form) and then drop the alias so the request doesn't ship two representations of the same intent past this point. - `validate_{chat_,}completion_fields_generic` now take `&mut` so they can call `normalize_return_token_ids` before `validate`. The `n == 1` constraint enforced by `validate_completion_token_ids_single_choice` inspects `nvext.extra_fields`, so the fold has to happen first. - The OpenAI Responses handler (which calls `chat_request.validate()` directly, not through the `_generic` helper) gets the same fold inline, right after `normalize_chat_reasoning_template_args`. - Callers passed `&request` -> `&mut request` (2 production sites in `http/service/openai.rs`, and 15 test sites updated in bulk). ## Tests 8 new unit tests, mirroring the existing nvext-path tests: - deserialization: field lives at request root, doesn't land in unsupported_fields - normalize folds `Some(true)` into extra_fields and drops the alias; validate() then succeeds - idempotent when both alias and nvext are set (single entry, not two) - `false` / `None` are no-ops (must not populate extra_fields) - `n > 1` with the alias is rejected by validate() with the same error message as the existing nvext path Rustfmt: clean. ## Not in scope - Response shape stays on the existing `choices[].nvext.completion_token_ids` path (merged in ai-dynamo#9649). Strict vLLM response parity (`choices[].token_ids` at top level) is a separate ask. - No RL admin plane, no weight-transfer, no new endpoints. This is only the request-side field parity split off from the closed ai-dynamo#9131/ai-dynamo#9382. Closes / relates to the request-side half of the intent expressed in ai-dynamo#9131 and ai-dynamo#9382. `nvext.token_data` / TITO passthrough (the other half) already merged in ai-dynamo#9649.

Add `return_token_ids: Option<bool>` to `NvCreateChatCompletionRequest` and `NvCreateCompletionRequest`, a convenience alias for `nvext.extra_fields = ["completion_token_ids"]`. ## Why vLLM's OpenAI extension of the same name (`return_token_ids: true` in `extra_body`) is what clients like the `speculators` `data_generation_offline.py` capture flow and Prime-RL send today. A request that works against a raw vLLM server currently fails against a Dynamo-fronted vLLM server: ``` Validation: Unsupported parameter(s): `return_token_ids` ``` because the field lands in `unsupported_fields` (catch-all) and `validate_no_unsupported_fields` rejects it. This PR closes that gap without changing the response-side path — the existing nvext machinery already emits `completion_token_ids` on the response. ## What changes - Two new `Option<bool>` fields, one per request struct, with `#[serde(default, skip_serializing_if = "Option::is_none")]`. - Two `normalize_return_token_ids(&mut self)` methods that fold `Some(true)` into `nvext.extra_fields.push("completion_token_ids")` (idempotent — no duplicate entry if the caller also set the nvext form) and then drop the alias so the request doesn't ship two representations of the same intent past this point. - `validate_{chat_,}completion_fields_generic` now take `&mut` so they can call `normalize_return_token_ids` before `validate`. The `n == 1` constraint enforced by `validate_completion_token_ids_single_choice` inspects `nvext.extra_fields`, so the fold has to happen first. - The OpenAI Responses handler (which calls `chat_request.validate()` directly, not through the `_generic` helper) gets the same fold inline, right after `normalize_chat_reasoning_template_args`. - Callers passed `&request` -> `&mut request` (2 production sites in `http/service/openai.rs`, and 15 test sites updated in bulk). ## Tests 8 new unit tests, mirroring the existing nvext-path tests: - deserialization: field lives at request root, doesn't land in unsupported_fields - normalize folds `Some(true)` into extra_fields and drops the alias; validate() then succeeds - idempotent when both alias and nvext are set (single entry, not two) - `false` / `None` are no-ops (must not populate extra_fields) - `n > 1` with the alias is rejected by validate() with the same error message as the existing nvext path Rustfmt: clean. ## Not in scope - Response shape stays on the existing `choices[].nvext.completion_token_ids` path (merged in ai-dynamo#9649). Strict vLLM response parity (`choices[].token_ids` at top level) is a separate ask. - No RL admin plane, no weight-transfer, no new endpoints. This is only the request-side field parity split off from the closed ai-dynamo#9131/ai-dynamo#9382. Closes / relates to the request-side half of the intent expressed in ai-dynamo#9131 and ai-dynamo#9382. `nvext.token_data` / TITO passthrough (the other half) already merged in ai-dynamo#9649. Signed-off-by: Yiqiu Liu <yiqiuliu@amazon.com>

Close the gap that makes a `return_token_ids: true` request work against a raw vLLM server but fail against a Dynamo-fronted vLLM server, and surface the two response fields a client using that flag actually needs. ## Request side Add `return_token_ids: Option<bool>` to `NvCreateChatCompletionRequest` and `NvCreateCompletionRequest`, a convenience alias for `nvext.extra_fields = ["completion_token_ids"]`. vLLM's OpenAI extension of the same name (`return_token_ids: true` in `extra_body`) is what clients like the `speculators` `data_generation_offline.py` capture flow and Prime-RL send today. Without the alias, requests fail with: ``` Validation: Unsupported parameter(s): `return_token_ids` ``` because the field lands in `unsupported_fields` (catch-all) and `validate_no_unsupported_fields` rejects it. - Two new `Option<bool>` fields, one per request struct, with `#[serde(default, skip_serializing_if = "Option::is_none")]`. - Two `normalize_return_token_ids(&mut self)` methods fold `Some(true)` into `nvext.extra_fields.push("completion_token_ids")` (idempotent) and drop the alias so the request doesn't ship two representations past this point. - `validate_{chat_,}completion_fields_generic` now take `&mut` and call `normalize_return_token_ids` before `validate`. The `n == 1` constraint enforced by `validate_completion_token_ids_single_choice` inspects `nvext.extra_fields`, so the fold has to happen first. - The OpenAI Responses handler (which calls `chat_request.validate()` directly) gets the same fold inline. ## Response side The two response fields a `return_token_ids: true` client typically needs alongside `completion_token_ids` are the prompt token ids echoed back and any `kv_transfer_params` from a disaggregated backend (e.g. the `hidden_states_path` from vLLM's KV-transfer connector). Add them under nvext, next to the existing `completion_token_ids`: - `NvExtResponse.prompt_token_ids: Option<Vec<TokenIdType>>` — sourced from `LLMEngineOutput.engine_data["prompt_token_ids"]` (the vLLM Python handler already populates this; other backends leave it absent, in which case nothing is emitted). - `NvExtResponse.kv_transfer_params: Option<serde_json::Value>` — sourced from `BackendOutput.disaggregated_params` (the canonical engine-owned location per its doc comment), falling back to `engine_data["kv_transfer_params"]`. Mirrors the pattern in `protocols/openai/generate.rs:534-560` for the native protocol. - Both fields are gated by the same `"completion_token_ids"` entry in `nvext.extra_fields` — so a request that sets `return_token_ids: true` (or the equivalent nvext form) gets the trio together. - Both are only emitted on the final chunk (when `finish_reason` is present) — there's no meaningful streaming semantic for a per-delta prompt-id list, and the vLLM handler surfaces both fields once at completion. - Wired through the shared `NvExtResponseFieldSelection::build_response_nvext` helper so chat and completions delta generators share one gating path. Note: fields land under `response.nvext`, matching Dynamo's existing extension convention (same layer as `completion_token_ids`). Clients that expect them at the response top level (as vLLM's own OpenAI server emits) need a small extractor to look under `nvext`. ## Tests Request side (8 tests, from the prior revision): - deserialization: field lives at request root, doesn't land in unsupported_fields - normalize folds `Some(true)` into extra_fields and drops the alias - idempotent when both alias and nvext are set (single entry, not two) - `false` / `None` are no-ops - `n > 1` with the alias is rejected Response side (4 new tests): - `from_nvext_completion_token_ids_gates_the_return_token_ids_trio` — a single `"completion_token_ids"` extra_fields entry turns on all three response fields (completion + prompt + kv_transfer_params) and does NOT bleed into unrelated selectors (timing, engine_data). - `build_response_nvext_prompt_token_ids_final_chunk_only` — absent mid-stream; present on the final chunk. - `build_response_nvext_kv_transfer_params_final_chunk_only` — same final-only shape, with a `hidden_states_path` payload. - `build_response_nvext_return_token_ids_trio_together` — end-to-end: request-side alias -> selector -> builder emits all three fields together on the final chunk. Existing 45 extensions unit tests continue to pass; existing 329 openai-protocol module tests continue to pass. ## Not in scope - Response fields at the response top level (as vLLM emits) — kept under `nvext` for consistency with Dynamo's existing extension layering; a client-side extractor is trivial. - No RL admin plane, no weight-transfer, no new endpoints. Closes / relates to the request-side + response-side halves of the intent expressed in ai-dynamo#9131 and ai-dynamo#9382. `nvext.token_data` / TITO passthrough (the third piece) already merged in ai-dynamo#9649. Signed-off-by: Yiqiu Liu <yiqiuliu@amazon.com>

biswapanda requested a review from a team May 16, 2026 00:03

pull-request-size Bot added the size/XL label May 16, 2026

github-actions Bot added feat frontend `python -m dynamo.frontend` and `dynamo-run in=http|text|grpc` labels May 16, 2026

biswapanda changed the title ~~feat(llm): add nvext tokens-in-tokens-out protocol~~ feat(llm): add nvext Tokens-in-Tokens-Out protocol May 16, 2026

biswapanda self-assigned this May 16, 2026

biswapanda enabled auto-merge (squash) May 16, 2026 00:05

coderabbitai Bot reviewed May 16, 2026

View reviewed changes

biswapanda changed the title ~~feat(llm): add nvext Tokens-in-Tokens-Out protocol~~ feat(RL): add nvext Tokens-in-Tokens-Out protocol May 16, 2026

biswapanda mentioned this pull request May 16, 2026

feat(RL): add vLLM Tokens-in-Tokens-Out and RL related response support #9651

Merged

dynamo-ops reviewed May 16, 2026

View reviewed changes

Comment thread lib/llm/src/protocols/openai/common_ext.rs Outdated

Comment thread lib/llm/src/protocols/openai/nvext.rs

Comment thread lib/llm/src/protocols/openai/completions/delta.rs Outdated

copy-pr-bot Bot temporarily deployed to GITLAB May 16, 2026 00:36 Inactive

biswapanda changed the title ~~feat(RL): add nvext Tokens-in-Tokens-Out protocol~~ feat(RL): add nvext Tokens-in-Tokens-Out and RL related response support May 16, 2026

biswapanda force-pushed the bis/nvext-tito-rl branch from 565fd7c to 2256c2a Compare May 16, 2026 01:12

copy-pr-bot Bot temporarily deployed to GITLAB May 16, 2026 01:12 Inactive

dynamo-ops reviewed May 16, 2026

View reviewed changes

Comment thread lib/llm/src/protocols/common/llm_backend.rs Outdated

Comment thread lib/llm/src/protocols/openai/completions.rs

dynamo-ops reviewed May 16, 2026

View reviewed changes

Comment thread lib/llm/src/protocols/common.rs Outdated

Comment thread lib/llm/src/preprocessor.rs

Comment thread lib/llm/src/backend.rs Outdated

copy-pr-bot Bot temporarily deployed to GITLAB May 16, 2026 01:55 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB May 16, 2026 02:18 Inactive

dynamo-ops reviewed May 16, 2026

View reviewed changes

Comment thread lib/llm/src/preprocessor.rs

copy-pr-bot Bot temporarily deployed to GITLAB May 16, 2026 05:18 Inactive

copy-pr-bot Bot temporarily deployed to GITLAB May 16, 2026 05:35 Inactive

biswapanda requested a review from dynamo-ops May 16, 2026 06:12

copy-pr-bot Bot temporarily deployed to GITLAB May 16, 2026 08:21 Inactive

biswapanda changed the title ~~feat(RL): add nvext Tokens-in-Tokens-Out and RL related response support~~ feat(RL): add nvext Tokens-in-Tokens-Out and RL related response support in frontend May 18, 2026

biswapanda changed the title ~~feat(RL): add nvext Tokens-in-Tokens-Out and RL related response support in frontend~~ feat(RL): add nvext Tokens-in-Tokens-Out and RL related response protocol and support it in frontend May 18, 2026

biswapanda changed the title ~~feat(RL): add nvext Tokens-in-Tokens-Out and RL related response protocol and support it in frontend~~ feat(RL): add nvext Tokens-in-Tokens-Out and RL related response protocol and frontend support May 18, 2026

richardhuo-nv approved these changes May 22, 2026

View reviewed changes

copy-pr-bot Bot temporarily deployed to GITLAB May 22, 2026 23:51 Inactive

biswapanda added 7 commits May 22, 2026 16:53

feat(llm): add nvext tokens-in-tokens-out protocol

b31e4fe

fix(llm): address nvext protocol review gaps

3bc7934

fix(llm): avoid prompt logprobs output shape break

650ac6f

Signed-off-by: Biswa Panda <biswa.panda@gmail.com>

fix(llm): keep backend sampling knobs out of public options

dcd3339

fix(llm): bound responses default output tokens

91fecee

fix(llm): preserve dynamic responses output cap

bce8eb2

fix(llm): move omitted max token policy to context

f46b9bf

biswapanda force-pushed the bis/nvext-tito-rl branch from 9f52a45 to f46b9bf Compare May 22, 2026 23:54

copy-pr-bot Bot temporarily deployed to GITLAB May 22, 2026 23:55 Inactive

biswapanda merged commit 85d0ea1 into main May 23, 2026
99 checks passed

biswapanda deleted the bis/nvext-tito-rl branch May 23, 2026 00:34

copy-pr-bot Bot had a problem deploying to GITLAB May 23, 2026 01:44 Failure

YiqiuLiu mentioned this pull request Aug 20, 2026

feat(openai): plumb return_token_ids end-to-end (request + response) #13588

Open

Navigation Menu

Uh oh!

feat(RL): add nvext Tokens-in-Tokens-Out and RL related response protocol and frontend support - #9649

feat(RL): add nvext Tokens-in-Tokens-Out and RL related response protocol and frontend support#9649
biswapanda merged 7 commits into
mainfrom
bis/nvext-tito-rl

biswapanda commented May 16, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented May 16, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot left a comment

Uh oh!

biswapanda commented May 16, 2026

Uh oh!

coderabbitai Bot commented May 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richardhuo-nv left a comment

Uh oh!

biswapanda commented May 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

biswapanda commented May 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Scope

Where To Review

Local Tests

Follow-up

Notes

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented May 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

biswapanda commented May 16, 2026

Uh oh!

coderabbitai Bot commented May 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

richardhuo-nv left a comment

Choose a reason for hiding this comment

Uh oh!

biswapanda commented May 22, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

biswapanda commented May 16, 2026 •

edited

Loading

coderabbitai Bot commented May 16, 2026 •

edited

Loading