refactor(relay): migrate dynamic plugin to switchyard-runner

CHANGELOG.md

@@ -8,6 +8,16 @@ adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 ### Added
 
+- **NeMo Relay native plugin** — a dynamically loaded integration that loads
+  Switchyard's standard TOML deployment and executes its named routes in
+  process. Managed calls require NeMo Relay 0.8.0 or newer; unknown models use
+  Relay's continuation unchanged.
+
+- **NeMo Relay routing marks** — routing-model usage, measured routing
+  overhead, and selected-model decisions are emitted as ATOF marks. The final
+  serving call remains represented only by Relay's outer LLM lifecycle event to
+  prevent double-counting.
+
 - **Advisor-gate routing** — new `advisor` route type pairing the serving
   executor with a stronger judge-only advisor that reviews terminal turns:
   APPROVE releases the buffered turn, REDO discards it and feeds the advisor's

Cargo.toml

@@ -9,6 +9,7 @@ members = [
     "crates/prefill-router",
     "crates/switchyard-py",
     "crates/protocol",
+    "crates/switchyard-nemo-relay-plugin",
     "crates/switchyard-runner",
     "crates/switchyard-server",
     "crates/switchyard-skill-distillation",
@@ -33,6 +34,7 @@ http = "1"
 httpdate = "1"
 jsonschema = { version = "0.49.4", default-features = false }
 jsonptr = { version = "0.8.1", default-features = false, features = ["std", "json", "resolve"] }
+nemo-relay-plugin = { git = "https://github.com/NVIDIA/NeMo-Relay.git", tag = "0.8.0-rc.3" }
 parking_lot = "0.12"
 rand = "0.10"
 regex = "1"

README.md

@@ -21,6 +21,7 @@ algorithm you write yourself.
 - **Protocol Translation**: convert between OpenAI Chat, Anthropic Messages, and OpenAI Responses formats
 - **Multi-Backend Routing**: random routing, LLM-as-classifier routing, signal-driven stage-router, or your own algorithm
 - **Operational Metrics**: Prometheus metrics cover requests, errors, latency, tokens, and routing overhead
+- **NeMo Relay Plugin**: run random, classifier, escalation, or stage routing in Relay while Switchyard owns provider HTTP dispatch
 
 ## Maturity
 
@@ -123,6 +124,7 @@ configured LLM client selects one upstream format.
 - **[`switchyard-libsy`](crates/libsy/README.md)**: embed routing algorithms in a Rust application
 - **[`switchyard-protocol`](crates/protocol/README.md)**: provider-neutral request, response, and streaming types
 - **[`switchyard-translation`](crates/switchyard-translation/README.md)**: request, response, and stream translation
+- **[`switchyard-nemo-relay-plugin`](crates/switchyard-nemo-relay-plugin/README.md)**: install Switchyard as a native NeMo Relay plugin
 
 ## Community
 

crates/switchyard-nemo-relay-plugin/Cargo.toml

@@ -0,0 +1,32 @@
+# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+[package]
+name = "switchyard-nemo-relay-plugin"
+version.workspace = true
+description = "Switchyard-owned HTTP routing plugin for NeMo Relay"
+authors.workspace = true
+edition.workspace = true
+license.workspace = true
+repository.workspace = true
+rust-version.workspace = true
+publish = false
+
+[lib]
+crate-type = ["cdylib"]
+
+[dependencies]
+futures-util.workspace = true
+http.workspace = true
+nemo-relay-plugin.workspace = true
+serde.workspace = true
+serde_json.workspace = true
+switchyard-llm-client.workspace = true
+switchyard-protocol.workspace = true
+switchyard-runner.workspace = true
+switchyard-translation.workspace = true
+tokio.workspace = true
+toml = "1.1"
+
+[dev-dependencies]
+tempfile = "3"

crates/switchyard-nemo-relay-plugin/README.md

@@ -0,0 +1,118 @@
+# Switchyard NeMo Relay Plugin
+
+`switchyard-nemo-relay-plugin` is a native NeMo Relay dynamic plugin. It loads
+a standard Switchyard TOML deployment from a file or Relay's nested plugin
+configuration and executes its configured routes through `switchyard-runner`.
+
+The plugin does not define a second routing or target configuration language.
+`switchyard-server` and Relay therefore use the same targets, client pooling,
+algorithm construction, retry policy, and route validation.
+
+## Install
+
+Build the platform bundle with the package script, then configure Relay to load
+the generated `relay-plugin.toml` manifest. The plugin requires NeMo Relay
+`>=0.8.0,<1.0`.
+
+## Configure Relay
+
+Use exactly one Switchyard deployment source. To share an existing deployment
+file with `switchyard-server`, configure its path:
+
+```toml
+[[plugins.dynamic]]
+manifest = "./plugins/switchyard/relay-plugin.toml"
+
+[plugins.dynamic.config]
+priority = 0
+switchyard_config_path = "/etc/switchyard/routes.toml"
+```
+
+`switchyard_config_path` is a Switchyard version-1 TOML deployment, accepted by both
+`switchyard-server` and `switchyard-runner`. See the
+[server configuration guide](../switchyard-server/CONFIGURATION.md) for the
+deployment schema and routing algorithms.
+
+To keep the deployment in the Relay configuration, nest the same version-1
+Switchyard configuration under `switchyard_config`:
+
+```toml
+[[plugins.dynamic]]
+manifest = "./plugins/switchyard/relay-plugin.toml"
+
+[plugins.dynamic.config]
+priority = 0
+
+[plugins.dynamic.config.switchyard_config]
+schema_version = 1
+
+[plugins.dynamic.config.switchyard_config.llm_clients.primary]
+format = "openai_chat"
+base_url = "https://example.test/v1"
+
+[plugins.dynamic.config.switchyard_config.targets.default]
+id = "example/model"
+llm_client = "primary"
+
+[plugins.dynamic.config.switchyard_config.routes.default]
+id = "switchyard/default"
+type = "passthrough"
+target = "default"
+```
+
+## Request handling
+
+For OpenAI Chat Completions, OpenAI Responses, and Anthropic Messages calls,
+the plugin decodes the Relay request and checks the requested model against the
+deployment's route IDs.
+
+- A configured route is executed by `switchyard-runner`.
+- An unknown model calls Relay's continuation unchanged.
+- The returned provider response is encoded back into the caller's wire format.
+- Streaming responses are returned as unpolled translated streams; Relay owns
+  cancellation and the outer serving-call lifecycle.
+
+Each route's target client must use the caller's wire format: `openai_chat`,
+`openai_responses`, or `anthropic_messages`. The runner selects the upstream
+backend from that format rather than translating a route to a different
+provider API. When one upstream model must serve multiple caller formats,
+declare a target and route for each corresponding client format.
+
+The plugin emits a routing request mark, routing-model call marks, measured
+routing-overhead marks, and a selected-model decision mark. Token usage is
+emitted as Switchyard metrics for both routing-model and answer-model calls;
+Relay retains ownership of the outer LLM lifecycle.
+
+## Observability
+
+When Relay is configured with OTLP logs and metrics exporters, the plugin emits
+typed telemetry through Relay's native plugin runtime:
+
+- Routing request, decision, and overhead marks are Info logs.
+- Per-routing-model call marks are Debug logs, including their outcome and
+  latency, but not token usage.
+- Terminal routing and response-finalization failures are Error logs. Their
+  payload contains only the safe Switchyard failure summary; it excludes
+  provider response bodies and free-form provider messages.
+- Metrics use bounded attributes only: algorithm for
+  `switchyard.routing.requests`; outcome for `switchyard.routing.llm_calls`
+  and `switchyard.routing.llm_call.duration`;
+  and safe failure kind, category, phase, and optional upstream HTTP status for
+  `switchyard.routing.failures`. `switchyard.routing.overhead` records total
+  routing latency, including routing-model calls; durations use milliseconds.
+- `switchyard.routing.llm_tokens` records normalized token usage with
+  `call_role` (`routing` or `answer`), configured `target_model`, and
+  `token_type` attributes. A provider may omit usage for streaming responses;
+  the plugin does not synthesize zero-value measurements.
+
+The plugin does not attach sessions, requests, or provider messages as metric
+attributes. `target_model` comes from the configured Switchyard target set,
+rather than arbitrary caller input, keeping the metric cardinality bounded by
+the deployment.
+
+## Failure policy
+
+`switchyard-llm-client` owns provider retry and route-candidate fallback
+behavior. The plugin does not maintain a separate trusted-default target or
+rerun routing after an execution failure. Failures outside the shared runner,
+including response translation failures, are returned to Relay.

crates/switchyard-nemo-relay-plugin/config.schema.json

@@ -0,0 +1,40 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "title": "Switchyard NeMo Relay Plugin",
+  "description": "Runs a Switchyard deployment through its shared in-process runner.",
+  "type": "object",
+  "additionalProperties": false,
+  "oneOf": [
+    { "required": ["switchyard_config_path"] },
+    { "required": ["switchyard_config"] }
+  ],
+  "properties": {
+    "priority": {
+      "type": "integer",
+      "default": 0,
+      "description": "NeMo Relay execution-intercept priority."
+    },
+    "executor": {
+      "type": "object",
+      "additionalProperties": false,
+      "description": "Optional NeMo Relay SDK executor override.",
+      "properties": {
+        "worker_threads": {
+          "type": "integer",
+          "minimum": 1,
+          "description": "Number of worker threads for the plugin's SDK-owned executor."
+        }
+      }
+    },
+    "switchyard_config_path": {
+      "type": "string",
+      "minLength": 1,
+      "description": "Path to a Switchyard version-1 TOML deployment shared with switchyard-server."
+    },
+    "switchyard_config": {
+      "type": "object",
+      "minProperties": 1,
+      "description": "Inline Switchyard version-1 deployment using the same schema as switchyard-server."
+    }
+  }
+}