fix(router): propagate trace context through worker dispatch

[wei-core]

Overview

Preserve the frontend OpenTelemetry trace when the KV push router dispatches a request to the selected worker.

Summary

• make kv_router.route_request an always-on request-plane span
• explicitly parent the route span to the captured distributed OTel context
• preserve trace flags, tracestate, and Dynamo request identifiers across the dispatch boundary
• add a regression test that captures a real ingress context, leaves the ambient span, and verifies outbound propagation

Details

The route span previously relied on the ambient tracing parent. That parent is not reliable across the KV-router dispatch boundary, so downstream request-plane header injection could start a separate trace.

The new helper records the complete distributed context on the route span and calls set_parent before instrumenting dispatch_exact or direct. The test exercises the detached-parent case with an unsampled W3C context and checks traceparent, tracestate, x-request-id, and request-id.

Where should the reviewer start?

lib/llm/src/kv_router/push_router.rs, especially route_request_span and route_request_span_propagates_distributed_trace_context.

Validation

• cargo fmt --all -- --check
• cargo metadata --locked --format-version 1 in ., lib/bindings/python, lib/runtime/examples, and lib/bindings/kvbm
• cargo test --locked -p dynamo-llm --no-default-features route_request_span_propagates_distributed_trace_context -- --nocapture — 1 passed
• cargo clippy --locked -p dynamo-llm --no-default-features --lib --tests -- -D warnings
• Ubuntu validation: https://github.com/wei-core/dynamo/actions/runs/32709956057

Related Issues

• Closes OTel trace context is not propagated through KV router dispatch #13345

---

Summary by CodeRabbit

• Improvements
  • Enhanced distributed request tracing for KV routing operations.
  • Trace context now carries across services, making request flows easier to follow.
  • Added richer routing metadata, including request, worker, and processing phase details.
  • Improved visibility into request propagation and outbound trace information for troubleshooting and performance analysis.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[github-actions]

👋 Hi wei-core! Thank you for contributing to ai-dynamo/dynamo.

Just a reminder: The NVIDIA Test Github Validation CI runs an essential subset of the testing framework to quickly catch errors.Your PR reviewers may elect to test the changes comprehensively before approving your changes.

🚀

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 0d6a9768-bfb4-4ed4-a2a0-8486c6268985

📥 Commits

Reviewing files that changed from the base of the PR and between 004cd02 and 5eb65d8.

⛔ Files ignored due to path filters (1)

• Cargo.lock is excluded by !**/*.lock

📒 Files selected for processing (2)

• lib/llm/Cargo.toml
• lib/llm/src/kv_router/push_router.rs

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The KV router now creates parented OpenTelemetry request spans and propagates distributed trace context to selected workers. Tests verify trace headers, request identifiers, and child span creation.

Changes

KV router tracing

Layer / File(s)	Summary
Create parented route spans lib/llm/Cargo.toml, lib/llm/src/kv_router/push_router.rs	Adds the tracing-opentelemetry dependency and creates kv_router.route_request spans with routing metadata and extracted parent context.
Propagate context through dispatch lib/llm/src/kv_router/push_router.rs	Uses the new span during cancellation-aware dispatch. Tests verify outbound trace headers, request IDs, external request identifiers, and child span IDs.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 5eb65

This localized tracing change preserves request context across worker dispatch and includes targeted validation; no actionable merge-blocking risk remains after normal checks and review.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes satisfy issue #13345 by parenting the route span to distributed context and testing downstream trace header propagation.
Out of Scope Changes check	✅ Passed	The dependency update, tracing changes, and regression test are directly related to the linked issue objectives.
Docstring Coverage	✅ Passed	Docstring check was indeterminate for this PR — some files could not be analyzed in time. Not blocking.
Description check	✅ Passed	The description includes all required sections and clearly documents the change, review starting points, validation, and linked issue.
Title check	✅ Passed	The title clearly and concisely describes the main change: propagating trace context through worker dispatch.

✨ Finishing Touches 💡 1

🛠️ Fix failing CI checks 💡

• Create stacked PR
• Commit on current branch

---

_{Comment @coderabbitai help to get the list of available commands.}

[devin-ai-integration]

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.