diff --git a/README.md b/README.md index bd06c9ac..2587a484 100644 --- a/README.md +++ b/README.md @@ -57,6 +57,7 @@ their models. | llama.cpp | `llamacpp` | Local (or remote) llama-server | | oMLX | `omlx` | Local oMLX server on Apple Silicon | | vLLM | `vllm` | Local or self-hosted vLLM server | +| MTPLX | `mtplx` | Local MTPLX server on Apple Silicon | | OpenAI-compatible | `openai-compatible` | Any endpoint that speaks the OpenAI API — set the base URL and key | Adding one that isn't here is a data change, not code — see @@ -80,7 +81,8 @@ model refs and quirks, and none of it is written down where you need it. it configures the agent for you: - **One command, any model.** Pick from a built-in catalogue — OpenRouter, - Bedrock, Ollama, llama.cpp, vLLM, oMLX, or any OpenAI-compatible endpoint. No + Bedrock, Ollama, llama.cpp, vLLM, oMLX, MTPLX, or any OpenAI-compatible + endpoint. No URLs to look up, and `spinloop list --models` fetches the model ids straight from the provider. - **Your config survives.** Settings are merged *into* what you already have. @@ -353,8 +355,9 @@ the bash and zsh completions for you. Running a model locally? `spinloop serve` reads a `Spinloop` and launches the inference server its `PROVIDER` names — `llamacpp` runs `llama-server`, `omlx` -runs [oMLX](https://omlx.ai) on Apple Silicon — so the same file that points -opencode at a model can start it too. The simple case needs no preset: +runs [oMLX](https://omlx.ai) and `mtplx` runs [MTPLX](https://mtplx.com), the +two Apple-Silicon engines — so the same file that points opencode at a model can +start it too. The simple case needs no preset: ```dockerfile # Spinloop @@ -376,8 +379,8 @@ the chosen section into the command instead — with anything the `Spinloop` sta (like `CONTEXT`) overriding the preset. It's the missing piece presets don't cover: launching a *single* model. `CONTEXT` always means the context per request; add `PARALLEL` to run more than one slot and `spinloop` works out each -engine's own accounting (llama.cpp's `--ctx-size` gets scaled, vLLM's and -oMLX's don't) — see +engine's own accounting (llama.cpp's `--ctx-size` gets scaled; vLLM's and +MTPLX's don't, and oMLX has none) — see [Parallelism](docs/commands/serve.md#parallelism) for the full mapping. Details in [`docs/commands/serve.md`](docs/commands/serve.md). @@ -582,8 +585,8 @@ machine; it is never sent to the deployed instance. Each provider declares which environment variable holds its key (`spinloop list` shows them). Values are looked up in your shell environment first, then a `.env` beside the `Spinloop`, so an exported variable always wins and the `.env` -only fills a gap. Local providers like Ollama, llama.cpp and oMLX need no -key; +only fills a gap. Local providers like Ollama, llama.cpp, oMLX and MTPLX need +no key; Bedrock authenticates through your AWS credentials. `spinloop harness` carries that same local environment to the agent it launches: @@ -603,7 +606,7 @@ SPINLOOP_BASE_URL=https://gateway/v1 spinloop add -p openai-compatible -m my-mod The flag wins over the env var, and either wins over the catalogue's defaults and the per-provider variables (`OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`, -`OMLX_BASE_URL`, `VLLM_BASE_URL`, `OPENAI_BASE_URL`). +`OMLX_BASE_URL`, `VLLM_BASE_URL`, `MTPLX_BASE_URL`, `OPENAI_BASE_URL`). ## Guides @@ -616,6 +619,7 @@ with a ready-to-apply `Spinloop`: - [Gemma-4-12B-IT on llama.cpp](examples/llamacpp/gemma4/README.md) - [Qwen3.6-35B-A3B on oMLX (Apple Silicon)](examples/omlx/qwen3.6/README.md) - [Gemma-4-E2B on oMLX (Apple Silicon)](examples/omlx/gemma-4-e2b/README.md) +- [Qwen3.8-27B on MTPLX (Apple Silicon)](examples/mtplx/qwen3.8-27b/README.md) - [Fetching a Spinloop from a URL](examples/remote-spinloop/README.md) ## Adding providers and models diff --git a/cmd/spinloop/remote.go b/cmd/spinloop/remote.go index 0d9887bd..148dfaf3 100644 --- a/cmd/spinloop/remote.go +++ b/cmd/spinloop/remote.go @@ -1050,6 +1050,21 @@ func runnerFor(provider string) (string, error) { } } +// nodeRunnerFor is the runner resolver for the node path — waking a fleet node +// that already exists. It accepts every engine `serve` can run and a daemon can +// supervise: llamacpp, vllm, and mtplx. MTPLX is Apple-Silicon-only and has no +// machine image, so it never becomes a cloud runner (see runnerFor). +func nodeRunnerFor(provider string) (string, error) { + switch provider { + case "llamacpp", "vllm", "mtplx": + return provider, nil + default: + return "", fmt.Errorf( + "PROVIDER %q cannot be woken: a fleet node runs a self-hosted engine, so use llamacpp, vllm or mtplx", + provider) + } +} + // splitModelQuant splits a model reference into the Hugging Face repo and an // optional quant tag, as used by llama.cpp's -hf (org/model:QUANT). Repo ids // cannot contain a colon, so the first one separates them. @@ -1072,6 +1087,12 @@ var cloudOwnedFlags = map[string]bool{ "hf-repo": true, "hf-file": true, "hf-token": true, "api-key": true, "api-key-file": true, "ctx-size": true, "alias": true, "metrics": true, + // MTPLX's own spellings of the settings the destination computes from the + // deploy config: the served name, the context window, and the slot count. + // Keyed by canonical name like the rest, so a preset's raw values do not + // double-define the computed flags. They collide with no llama.cpp or vLLM + // key, so the cloud path is untouched. + "model-id": true, "context-window": true, "max-active-requests": true, // Companion weights: the cloud syncs these from S3 and names them at its // own paths, so the preset's local paths must not travel. Only the // location is cloud-owned — how the engine is asked to use a drafter @@ -1111,6 +1132,35 @@ func parallelPresetKey(runner string) string { return "max-num-seqs" case "omlx": return "max-concurrent-requests" + case "mtplx": + return "max-active-requests" + default: + return "" + } +} + +// modelPresetKey names the preset key holding the model, in that runner's own +// vocabulary — the same split the *ServeParams functions make when they read +// it. A runner with no preset model key yields "", which reads as "not set". +func modelPresetKey(runner string) string { + switch runner { + case "mtplx": + return "model" + case "llamacpp", "vllm": + return "hf" + default: + return "" + } +} + +// contextPresetKey names the preset key holding the context window, in that +// runner's own vocabulary. +func contextPresetKey(runner string) string { + switch runner { + case "mtplx": + return "context-window" + case "llamacpp", "vllm": + return "ctx-size" default: return "" } @@ -1157,6 +1207,7 @@ func dropOwned(owned func(string) bool, params []preset.Param) []preset.Param { // caller where that is not true. func deployConfigFor(sel spinloop.Selection, spinloopPath string) (remote.DeployConfig, error) { return deployConfig(sel, spinloopPath, deployTarget{ + runner: runnerFor, requireContext: true, seedsWeights: true, owns: isCloudOwned, @@ -1170,13 +1221,18 @@ func deployConfigFor(sel spinloop.Selection, spinloopPath string) (remote.Deploy // that `spinloop serve` runs happily needs no CONTEXT added merely to be routed; // and the preset's bind survives, so an engine told to listen on 0.0.0.0 does. func deployConfigForNode(sel spinloop.Selection, spinloopPath string) (remote.DeployConfig, error) { - return deployConfig(sel, spinloopPath, deployTarget{owns: isNodeOwned}) + return deployConfig(sel, spinloopPath, deployTarget{ + runner: nodeRunnerFor, + owns: isNodeOwned, + }) } -// deployTarget is what the derivation cannot decide for itself: whether a -// context size is required, which preset flags the destination assigns, and -// whether it fetches the weights itself (and so needs companions named). +// deployTarget is what the derivation cannot decide for itself: which runners +// it accepts, whether a context size is required, which preset flags the +// destination assigns, and whether it fetches the weights itself (and so needs +// companions named). type deployTarget struct { + runner func(provider string) (string, error) requireContext bool seedsWeights bool owns func(key string) bool @@ -1185,7 +1241,7 @@ type deployTarget struct { func deployConfig(sel spinloop.Selection, spinloopPath string, target deployTarget) (remote.DeployConfig, error) { var dc remote.DeployConfig - runner, err := runnerFor(sel.Provider) + runner, err := target.runner(sel.Provider) if err != nil { return dc, err } @@ -1219,14 +1275,19 @@ func deployConfig(sel spinloop.Selection, spinloopPath string, target deployTarg model := sel.Model if model == "" { - model = presetValue("hf", global, params) + model = presetValue(modelPresetKey(dc.Runner), global, params) } if model == "" { return dc, fmt.Errorf( - "nothing to deploy: set MODEL (an HF repo like org/model:QUANT) in %s, or hf in its preset", - spinloopPath) - } - if isModelPath(model) { + "nothing to deploy: set MODEL in %s, or %s in its preset", + spinloopPath, modelPresetKey(dc.Runner)) + } + // A local model path is refused only where the destination fetches the + // weights itself — the cloud, which cannot ship a file. A node has the file + // the Spinloop points at, so the node path carries the path as the model to + // load; today's unconditional check also blocked llamacpp and vllm node + // wakes with local weights, which moving it unblocks too. + if target.seedsWeights && isModelPath(model) { return dc, fmt.Errorf( "cannot deploy the local model file %q: the cloud downloads weights from Hugging Face, so name a repo (org/model:QUANT)", model) @@ -1235,10 +1296,10 @@ func deployConfig(sel spinloop.Selection, spinloopPath string, target deployTarg context := sel.Context if context == "" { - context = presetValue("ctx-size", global, params) + context = presetValue(contextPresetKey(dc.Runner), global, params) } if context == "" && target.requireContext { - return dc, fmt.Errorf("no context size: set CONTEXT in %s, or ctx-size in its preset", spinloopPath) + return dc, fmt.Errorf("no context size: set CONTEXT in %s, or %s in its preset", spinloopPath, contextPresetKey(dc.Runner)) } if context != "" { n, err := contextsize.Parse(context) diff --git a/cmd/spinloop/serve.go b/cmd/spinloop/serve.go index 94ef5a38..500dd436 100644 --- a/cmd/spinloop/serve.go +++ b/cmd/spinloop/serve.go @@ -12,6 +12,7 @@ import ( "net/url" "os" "os/exec" + "sort" "strconv" "strings" @@ -35,6 +36,10 @@ var omlxBinary = "" // tests can point it at a stub instead of a real install. var vllmBinary = "vllm" +// mtlxBinary is the MTPLX executable that `serve` launches. A package var so +// tests can point it at a stub instead of a real install. +var mtlxBinary = "mtplx" + // omlxBundleBinary is where the macOS app installs its CLI. oMLX ships as a // signed app rather than a PATH install, so a user who has only ever launched // it from the menu bar still has this and nothing on their PATH. @@ -95,56 +100,92 @@ type serveEngine struct { positional func(sel spinloop.Selection) []string } +// engines is the set of local inference servers `serve` can launch, keyed by +// the PROVIDER that selects one. It is the single source of truth for what +// serve runs: the error for an unservable PROVIDER and the help text both name +// the engines from here rather than from a list written out by hand. +var engines = map[string]serveEngine{ + "llamacpp": { + binary: func() string { return llamaServerBinary }, + dialect: preset.LlamaCpp, + params: llamacppServeParams, + needsModel: true, + installHint: "install llama.cpp (e.g. brew install llama.cpp) or check the path", + metricsArgs: []string{"--metrics"}, + metricsEngine: "llamacpp", + apiKeyFileFlag: "--api-key-file", + defaultBaseURL: "http://127.0.0.1:8080", + defaultBindLoopback: true, + }, + "omlx": { + binary: resolveOMLXBinary, + subcommand: []string{"serve"}, + dialect: preset.OMLX, + params: omlxServeParams, + installHint: "install oMLX (https://omlx.ai) or check the path", + }, + "mtplx": { + binary: func() string { return mtlxBinary }, + subcommand: []string{"serve"}, + dialect: preset.MTPLX, + params: mtplxServeParams, + needsModel: true, + installHint: "install MTPLX (https://mtplx.com) or check the path", + // MTPLX has no Prometheus endpoint to scrape, so there is no + // metrics switch to append and no dialect for the scraper. + apiKeyFileFlag: "--api-key-file", + defaultBaseURL: "http://127.0.0.1:8000", + defaultBindLoopback: true, + }, + "vllm": { + binary: func() string { return vllmBinary }, + subcommand: []string{"serve"}, + dialect: preset.VLLM, + params: vllmServeParams, + needsModel: true, + installHint: "install vLLM (pip install vllm) or check the path", + // vLLM serves /metrics unconditionally, so no switch to append. + metricsEngine: "vllm", + defaultBaseURL: "http://127.0.0.1:8000", + positional: func(sel spinloop.Selection) []string { + if sel.Model == "" { + return nil + } + return []string{sel.Model} + }, + }, +} + // engineFor maps a Spinloop's PROVIDER to the engine `serve` launches locally. // It is the local twin of runnerFor: PROVIDER already names the engine, so no // separate keyword is needed. Providers that are not self-hosted engines have // nothing to launch. func engineFor(provider string) (serveEngine, error) { - switch provider { - case "llamacpp": - return serveEngine{ - binary: func() string { return llamaServerBinary }, - dialect: preset.LlamaCpp, - params: llamacppServeParams, - needsModel: true, - installHint: "install llama.cpp (e.g. brew install llama.cpp) or check the path", - metricsArgs: []string{"--metrics"}, - metricsEngine: "llamacpp", - apiKeyFileFlag: "--api-key-file", - defaultBaseURL: "http://127.0.0.1:8080", - defaultBindLoopback: true, - }, nil - case "omlx": - return serveEngine{ - binary: resolveOMLXBinary, - subcommand: []string{"serve"}, - dialect: preset.OMLX, - params: omlxServeParams, - installHint: "install oMLX (https://omlx.ai) or check the path", - }, nil - case "vllm": - return serveEngine{ - binary: func() string { return vllmBinary }, - subcommand: []string{"serve"}, - dialect: preset.VLLM, - params: vllmServeParams, - needsModel: true, - installHint: "install vLLM (pip install vllm) or check the path", - // vLLM serves /metrics unconditionally, so no switch to append. - metricsEngine: "vllm", - defaultBaseURL: "http://127.0.0.1:8000", - positional: func(sel spinloop.Selection) []string { - if sel.Model == "" { - return nil - } - return []string{sel.Model} - }, - }, nil - default: + engine, ok := engines[provider] + if !ok { return serveEngine{}, fmt.Errorf( - "PROVIDER %q cannot be served locally: serve runs a self-hosted engine, so use llamacpp, omlx or vllm", - provider) + "PROVIDER %q cannot be served locally: serve runs a self-hosted engine, so use %s", + provider, orList(servedProviders())) + } + return engine, nil +} + +// servedProviders lists the PROVIDERs serve can launch, in stable order. +func servedProviders() []string { + names := make([]string, 0, len(engines)) + for name := range engines { + names = append(names, name) + } + sort.Strings(names) + return names +} + +// orList joins names the way an English sentence does: "a, b or c". +func orList(names []string) string { + if len(names) == 1 { + return names[0] } + return strings.Join(names[:len(names)-1], ", ") + " or " + names[len(names)-1] } // cmdServe reads a Spinloop and runs the inference server its PROVIDER names. @@ -163,11 +204,11 @@ func serveCmd() *cobra.Command { c := &cobra.Command{ Use: "serve", Short: "run the Spinloop's inference server", - Long: `runs the inference server the Spinloop's PROVIDER names — llamacpp -(llama-server) or omlx (Apple Silicon). With a PRESET it turns the matching -section into the command, reading it in that engine's flag vocabulary; -otherwise it derives one from the Spinloop's own instructions. Prints the -command before running it; --dry-run/-n prints without launching the server.`, + Long: fmt.Sprintf(`runs the inference server the Spinloop's PROVIDER names — %s. +With a PRESET it turns the matching section into the command, reading it in +that engine's flag vocabulary; otherwise it derives one from the Spinloop's +own instructions. Prints the command before running it; --dry-run/-n prints +without launching the server.`, orList(servedProviders())), Args: cobra.ArbitraryArgs, SilenceErrors: true, SilenceUsage: true, @@ -402,6 +443,52 @@ func omlxServeParams(sel spinloop.Selection) ([]preset.Param, error) { return append(params, bind...), nil } +// mtplxServeParams turns the MTPLX settings a Spinloop states into preset +// params: MODEL names the weights — an MTPLX-optimised HF repo or a local +// path, both taken verbatim by --model — and ALIAS, CONTEXT, and BASEURL fill +// in the served name, the context window, and the bind address. +// +// --download is always passed: MTPLX fetches an optimised pack it does not +// have, the same job -hf does for llama-server, and it is a no-op when the +// model is already present. +// +// PARALLEL maps to --max-active-requests, the cap on admitted concurrent +// requests. How admitted requests execute — the scheduler mode, serial or +// parallel — is a preset concern: PARALLEL never selects it. And like vLLM +// and oMLX there is no context flag to scale: --context-window bounds a +// single request, independently of how many are admitted. +func mtplxServeParams(sel spinloop.Selection) ([]preset.Param, error) { + var params []preset.Param + if sel.Model != "" { + params = append(params, preset.Param{Key: "model", Value: sel.Model}) + } + // MTPLX fetches an optimised model it does not have; a no-op when the + // model is already present. + params = append(params, preset.Param{Key: "download", Value: ""}) + if sel.Alias != "" { + params = append(params, preset.Param{Key: "model-id", Value: sel.Alias}) + } + if sel.Context != "" { + n, err := contextsize.Parse(sel.Context) + if err != nil { + return nil, err + } + params = append(params, preset.Param{Key: "context-window", Value: strconv.Itoa(n)}) + } + if sel.Parallel != "" { + n, err := parseParallel(sel.Parallel) + if err != nil { + return nil, err + } + params = append(params, preset.Param{Key: "max-active-requests", Value: strconv.Itoa(n)}) + } + bind, err := bindAddressParams(sel) + if err != nil { + return nil, err + } + return append(params, bind...), nil +} + // llamacppServeParams turns the llama-server settings a Spinloop states into // preset params: the provider-native MODEL supplies the model source (hf for a // Hugging Face repo, model for a .gguf path); ALIAS, CONTEXT, PARALLEL, and diff --git a/cmd/spinloop/serve_daemon.go b/cmd/spinloop/serve_daemon.go index 842e7c2f..5112900b 100644 --- a/cmd/spinloop/serve_daemon.go +++ b/cmd/spinloop/serve_daemon.go @@ -399,11 +399,12 @@ func withMetricsArgs(argv []string, engine serveEngine) []string { return append(argv, engine.metricsArgs...) } -// scrapeTargetFor locates the engine's own /metrics for the collector, with +// scrapeTargetFor locates the engine's own endpoint for the collector, with // the API key lifted from the command — a literal --api-key, or the contents // of an --api-key-file (how the cloud delivers it) — so a gated /metrics still -// answers. An engine with no metrics endpoint yields the zero target (no -// scrape). +// answers. The address is always resolved, even for an engine with no metrics +// dialect: the readiness check probes the engine's own /health there, and an +// engine with no /metrics to parse simply has an empty Engine. // // The address is taken from the engine's own --host/--port when it states // them, because that is where the process actually binds. Only then the @@ -414,9 +415,6 @@ func withMetricsArgs(argv []string, engine serveEngine) []string { // deploy config's --port 8000. Every scrape was refused, silently, and the // activity record — derived from those counters — never moved. func scrapeTargetFor(engine serveEngine, baseURL string, argv []string) metrics.ScrapeTarget { - if engine.metricsEngine == "" { - return metrics.ScrapeTarget{} - } bind := engineBindFrom(argv) if b := bindBaseURL(bind.host, bind.port); b != "" { baseURL = b diff --git a/cmd/spinloop/serve_daemon_test.go b/cmd/spinloop/serve_daemon_test.go index 4bc091c7..4203f867 100644 --- a/cmd/spinloop/serve_daemon_test.go +++ b/cmd/spinloop/serve_daemon_test.go @@ -777,6 +777,39 @@ func TestScrapeTargetForHonoursTheEngineBind(t *testing.T) { } } +// TestScrapeTargetForDialectlessEngineStillResolvesAnAddress covers mtplx: it +// has no /metrics to scrape, so the target's dialect is empty, but the address +// is still resolved from the bind, BASEURL, or default — the readiness check +// needs that address to probe /health even where there is nothing to scrape. +func TestScrapeTargetForDialectlessEngineStillResolvesAnAddress(t *testing.T) { + engine, err := engineFor("mtplx") + if err != nil { + t.Fatal(err) + } + + cases := []struct { + name string + baseURL string + argv []string + want string + }{ + {"stated bind wins", "", []string{"mtplx", "serve", "--host", "0.0.0.0", "--port", "9100"}, "http://127.0.0.1:9100"}, + {"BASEURL when no bind", "http://127.0.0.1:9000/v1", []string{"mtplx", "serve"}, "http://127.0.0.1:9000/v1"}, + {"engine default when nothing else", "", []string{"mtplx", "serve"}, "http://127.0.0.1:8000"}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + got := scrapeTargetFor(engine, tc.baseURL, tc.argv) + if got.BaseURL != tc.want { + t.Errorf("BaseURL = %q, want %q", got.BaseURL, tc.want) + } + if got.Engine != "" { + t.Errorf("Engine = %q, want empty (mtplx has no metrics dialect)", got.Engine) + } + }) + } +} + // The endpoint a node advertises to a router is derived from the same command // line the metrics scrape reads, so the two cannot disagree about one engine. func TestEngineEndpointFor(t *testing.T) { @@ -957,6 +990,157 @@ ctx-size = 4096 } } +// mtplxNodePreset is an MTPLX-vocabulary preset: long-form keys, the model under +// `model`, the window under `context-window`, the cap under +// `max-active-requests`, a served name under `model-id`, and a scheduling mode +// the engine — not spinloop — owns. +const mtplxNodePreset = ` +[*] +host = 0.0.0.0 +port = 8000 + +[qwen] +model = Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed +context-window = 32768 +scheduler-mode = parallel +max-active-requests = 4 +model-id = preset-alias +` + +// A fleet node runs mtplx, so the node path accepts it and reads the model, +// window, and slot count back out of an MTPLX preset — each in its own +// vocabulary — while the Spinloop's ALIAS supplies the served name and the +// operator's own settings (the bind, the scheduling mode) survive. +func TestNodeDeployConfigMtplxFromPreset(t *testing.T) { + dir := t.TempDir() + mustWrite(t, filepath.Join(dir, "preset.ini"), mtplxNodePreset) + spinloopPath := filepath.Join(dir, "Spinloop") + mustWrite(t, spinloopPath, "PROVIDER mtplx\nALIAS qwen\nPRESET ./preset.ini\n") + sel, _, err := readSpinloop("test", spinloopPath) + if err != nil { + t.Fatal(err) + } + + node, err := deployConfigForNode(sel, spinloopPath) + if err != nil { + t.Fatal(err) + } + if node.Runner != "mtplx" { + t.Errorf("runner = %q, want mtplx", node.Runner) + } + if node.ModelID != "Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed" || node.Quant != "" { + t.Errorf("model = %q quant = %q, want the preset's model with no quant", node.ModelID, node.Quant) + } + if node.ContextSize != 32768 { + t.Errorf("contextSize = %d, want the preset's context-window", node.ContextSize) + } + if node.Parallel != 4 { + t.Errorf("parallel = %d, want the preset's max-active-requests", node.Parallel) + } + // The served name comes from the Spinloop's ALIAS, not the preset's model-id. + if node.ServedModelName != "qwen" { + t.Errorf("servedModelName = %q, want the ALIAS, not the preset's model-id", node.ServedModelName) + } + + args := strings.Join(node.ServeArgs, " ") + // The daemon supplies these from the config, so the preset's raw values go. + for _, unwanted := range []string{"--model ", "--model-id", "--context-window", "--max-active-requests"} { + if strings.Contains(args, unwanted) { + t.Errorf("a node's serve args should not repeat %q, got: %s", unwanted, args) + } + } + // The operator's own settings survive: the bind, and the scheduling mode, + // which spinloop does not compute. + for _, want := range []string{"--host 0.0.0.0", "--port 8000", "--scheduler-mode parallel"} { + if !strings.Contains(args, want) { + t.Errorf("a node's serve args should keep %q, got: %s", want, args) + } + } +} + +// A node wake needs no preset: the Spinloop's own MODEL, CONTEXT, and PARALLEL +// drive the config, exactly as they do a local `serve`. +func TestNodeDeployConfigMtplxWithoutPreset(t *testing.T) { + spinloopPath := writeDeploySpinloop(t, + "PROVIDER mtplx\nMODEL Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed\nALIAS qwen\nCONTEXT 128k\nPARALLEL 2\n", "") + sel, _, err := readSpinloop("test", spinloopPath) + if err != nil { + t.Fatal(err) + } + node, err := deployConfigForNode(sel, spinloopPath) + if err != nil { + t.Fatal(err) + } + if node.Runner != "mtplx" || node.ModelID != "Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed" { + t.Errorf("runner = %q model = %q, want mtplx and the stated model", node.Runner, node.ModelID) + } + if node.ContextSize != 128000 { + t.Errorf("contextSize = %d, want the stated 128k", node.ContextSize) + } + if node.Parallel != 2 { + t.Errorf("parallel = %d, want the stated 2", node.Parallel) + } +} + +// A node has the file its Spinloop points at, so a local model path is carried +// as the model to load rather than refused. +func TestNodeDeployConfigMtplxLocalModelPath(t *testing.T) { + spinloopPath := writeDeploySpinloop(t, + "PROVIDER mtplx\nMODEL ./models/qwen-mtplx\nCONTEXT 32k\n", "") + sel, _, err := readSpinloop("test", spinloopPath) + if err != nil { + t.Fatal(err) + } + node, err := deployConfigForNode(sel, spinloopPath) + if err != nil { + t.Fatalf("a node wake with a local model path should succeed: %v", err) + } + if node.ModelID != "./models/qwen-mtplx" { + t.Errorf("modelId = %q, want the local path carried as the model", node.ModelID) + } +} + +// Moving the local-path check to the weights-seeding target unblocks llamacpp +// (and vllm) node wakes with local weights too, which the old unconditional +// check blocked. +func TestNodeDeployConfigLlamacppLocalModelPathNowSucceeds(t *testing.T) { + spinloopPath := writeDeploySpinloop(t, + "PROVIDER llamacpp\nMODEL ./models/qwen.gguf\nCONTEXT 32k\n", "") + sel, _, err := readSpinloop("test", spinloopPath) + if err != nil { + t.Fatal(err) + } + node, err := deployConfigForNode(sel, spinloopPath) + if err != nil { + t.Fatalf("a node wake with a local model path should succeed: %v", err) + } + if node.ModelID != "./models/qwen.gguf" { + t.Errorf("modelId = %q, want the local path", node.ModelID) + } + + // The cloud still refuses the same file: it downloads from Hugging Face. + sel.Context = "32k" + if _, err := deployConfigFor(sel, spinloopPath); err == nil || + !strings.Contains(err.Error(), "cannot deploy the local model file") { + t.Errorf("the cloud should still refuse a local model file, got %v", err) + } +} + +// MTPLX is Apple-Silicon-only and has no machine image, so it never becomes a +// cloud runner: the cloud path refuses it with its existing message. +func TestDeployConfigFor_CloudStillRefusesMtplx(t *testing.T) { + spinloopPath := writeDeploySpinloop(t, + "PROVIDER mtplx\nMODEL Youssofal/Qwen3.8-27B\nCONTEXT 32k\n", "") + sel, _, err := readSpinloop("test", spinloopPath) + if err != nil { + t.Fatal(err) + } + if _, err := deployConfigFor(sel, spinloopPath); err == nil || + !strings.Contains(err.Error(), "cannot be deployed") { + t.Errorf("the cloud should refuse mtplx, got %v", err) + } +} + // The bind reaching the engine is only half of it: the daemon must then report // that engine as reachable, or routing refuses a node that is in fact fine. func TestNodeBindReachesTheReportedEndpoint(t *testing.T) { diff --git a/cmd/spinloop/serve_test.go b/cmd/spinloop/serve_test.go index 480cfddc..348c9edf 100644 --- a/cmd/spinloop/serve_test.go +++ b/cmd/spinloop/serve_test.go @@ -56,6 +56,13 @@ func stubOMLX(t *testing.T, argsFile string) { stubServer(t, &omlxBinary, "omlx-cli", argsFile) } +// stubMTPLX does the same for the MTPLX CLI, so no MTPLX install is needed to +// pin the argv serve builds for it. +func stubMTPLX(t *testing.T, argsFile string) { + t.Helper() + stubServer(t, &mtlxBinary, "mtplx", argsFile) +} + // omlxPreset is a preset written in oMLX's own flag vocabulary. `m` is here // deliberately: llama.cpp's dialect would rewrite it to --model, and this preset // must not be read that way. @@ -80,6 +87,32 @@ func writeOMLXSpinloop(t *testing.T, spinloopBody string) string { return spinloopPath } +// mtplxDialectPreset is a preset written in MTPLX's own flag vocabulary. `c` +// is here deliberately: llama.cpp's dialect would rewrite it to --ctx-size, +// and this preset must not be read that way. +const mtplxDialectPreset = `[*] +host = 127.0.0.1 +port = 8000 + +[qwen] +model = Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed +context-window = 32768 +scheduler-mode = parallel +max-active-requests = 4 +c = should-stay-literal +` + +// writeMTPLXSpinloop writes an MTPLX preset.ini and a Spinloop referencing it +// into a fresh temp dir, and returns the Spinloop's path. +func writeMTPLXSpinloop(t *testing.T, spinloopBody string) string { + t.Helper() + dir := t.TempDir() + mustWrite(t, filepath.Join(dir, "preset.ini"), mtplxDialectPreset) + spinloopPath := filepath.Join(dir, "Spinloop") + mustWrite(t, spinloopPath, spinloopBody) + return spinloopPath +} + func TestCmdServe_PresetDryRun(t *testing.T) { spinloopPath := writePresetSpinloop(t, "PROVIDER llamacpp\nALIAS qwen\nPRESET preset.ini\n") @@ -399,9 +432,9 @@ func TestCmdServe_ParallelOverridesPresetValue(t *testing.T) { } // TestCmdServe_InvalidParallel checks a non-positive or non-numeric PARALLEL -// fails rather than being passed to the engine, for all three engines. +// fails rather than being passed to the engine, for all four engines. func TestCmdServe_InvalidParallel(t *testing.T) { - for _, provider := range []string{"llamacpp", "vllm", "omlx"} { + for _, provider := range []string{"llamacpp", "vllm", "omlx", "mtplx"} { for _, bad := range []string{"0", "-1", "abc"} { t.Run(provider+"/"+bad, func(t *testing.T) { dir := t.TempDir() @@ -671,6 +704,174 @@ func TestCmdServe_OMLXNotFound(t *testing.T) { } } +// TestCmdServe_MTPPLXDryRun pins the whole MTPLX shape: the `serve` +// subcommand, the model as --model, the served name as --model-id, the +// context window, the parallel cap, a bind address taken from BASEURL, and +// --download so a missing model is fetched by the engine. +func TestCmdServe_MTPPLXDryRun(t *testing.T) { + dir := t.TempDir() + spinloopPath := filepath.Join(dir, "Spinloop") + mustWrite(t, spinloopPath, + "PROVIDER mtplx\nMODEL Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed\nALIAS qwen\nCONTEXT 128k\nPARALLEL 2\nBASEURL http://127.0.0.1:9100/v1\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", spinloopPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + for _, want := range []string{ + "mtplx serve", + "--model Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed", + "--download", + "--model-id qwen", + "--context-window 128000", + "--max-active-requests 2", + "--host 127.0.0.1", + "--port 9100", + } { + if !strings.Contains(out, want) { + t.Errorf("command missing %q:\n%s", want, out) + } + } + // Another engine's vocabulary must not leak into the command. + for _, unwanted := range []string{"llama-server", "--hf-repo", "--ctx-size", "--max-num-seqs"} { + if strings.Contains(out, unwanted) { + t.Errorf("command should not contain %q:\n%s", unwanted, out) + } + } +} + +// TestCmdServe_MTPPLXDefaultsWithoutBaseURL checks that with no BASEURL no bind +// flag is emitted at all, so MTPLX's own defaults stand. +func TestCmdServe_MTPPLXDefaultsWithoutBaseURL(t *testing.T) { + dir := t.TempDir() + spinloopPath := filepath.Join(dir, "Spinloop") + mustWrite(t, spinloopPath, "PROVIDER mtplx\nMODEL org/model\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", spinloopPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + if strings.Contains(out, "--host") || strings.Contains(out, "--port") { + t.Errorf("no BASEURL should mean no bind flags:\n%s", out) + } + if !strings.Contains(out, "--download") { + t.Errorf("--download is always passed:\n%s", out) + } +} + +// TestCmdServe_MTPPLXNeedsModel checks that MTPLX, like llama.cpp and vLLM, +// will not start without being told what to load. +func TestCmdServe_MTPPLXNeedsModel(t *testing.T) { + dir := t.TempDir() + spinloopPath := filepath.Join(dir, "Spinloop") + mustWrite(t, spinloopPath, "PROVIDER mtplx\n") + if err := cmdServe([]string{"--dry-run", spinloopPath}); err == nil { + t.Error("expected error when there is neither a PRESET nor a MODEL") + } +} + +// TestCmdServe_MTPPLXNeverPassesAPIKey is a security guard: serve prints the +// command it runs, so a resolved secret must never reach it; a gated engine is +// gated with a key file the daemon writes, not a literal on the line. +func TestCmdServe_MTPPLXNeverPassesAPIKey(t *testing.T) { + t.Setenv("OPENAI_API_KEY", "sk-must-not-appear") + dir := t.TempDir() + spinloopPath := filepath.Join(dir, "Spinloop") + mustWrite(t, spinloopPath, "PROVIDER mtplx\nMODEL org/model\nBASEURL http://127.0.0.1:9100/v1\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", spinloopPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + if strings.Contains(out, "sk-must-not-appear") || strings.Contains(out, "--api-key") { + t.Errorf("serve must not pass or print an API key:\n%s", out) + } +} + +// TestCmdServe_MTPPLXPresetKeysPassThrough is the guard on the dialect split: +// an MTPLX preset is read in MTPLX's vocabulary, so a key llama.cpp would +// rewrite is left exactly as written, and every long-form key renders as its +// own flag. +func TestCmdServe_MTPPLXPresetKeysPassThrough(t *testing.T) { + spinloopPath := writeMTPLXSpinloop(t, "PROVIDER mtplx\nALIAS qwen\nPRESET preset.ini\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", spinloopPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + for _, want := range []string{ + "--model Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed", + "--context-window 32768", + "--scheduler-mode parallel", + "--max-active-requests 4", + "-c should-stay-literal", + "--download", + } { + if !strings.Contains(out, want) { + t.Errorf("command missing %q:\n%s", want, out) + } + } + if strings.Contains(out, "--ctx-size") { + t.Errorf("llama.cpp aliasing leaked into the MTPLX dialect:\n%s", out) + } +} + +// TestCmdServe_MTPPLXSpinloopOverridesPreset checks the Spinloop still wins +// over its preset for the settings it states, as it does for every engine. +func TestCmdServe_MTPPLXSpinloopOverridesPreset(t *testing.T) { + spinloopPath := writeMTPLXSpinloop(t, + "PROVIDER mtplx\nALIAS qwen\nPRESET preset.ini\nCONTEXT 128k\nPARALLEL 8\nBASEURL http://0.0.0.0:9999/v1\n") + + out := captureStdout(t, func() { + if err := cmdServe([]string{"--dry-run", spinloopPath}); err != nil { + t.Fatalf("cmdServe: %v", err) + } + }) + + for _, want := range []string{ + "--context-window 128000", + "--max-active-requests 8", + "--host 0.0.0.0", + "--port 9999", + } { + if !strings.Contains(out, want) { + t.Errorf("command missing %q:\n%s", want, out) + } + } + // The preset's values are overridden in place, not duplicated. + if strings.Contains(out, "--context-window 32768") || strings.Contains(out, "--max-active-requests 4") { + t.Errorf("preset values should have been overridden:\n%s", out) + } +} + +// TestCmdServe_MTPPLXNotFound checks the install hint names MTPLX rather than +// llama.cpp. +func TestCmdServe_MTPPLXNotFound(t *testing.T) { + orig := mtlxBinary + mtlxBinary = filepath.Join(t.TempDir(), "definitely-not-installed") + t.Cleanup(func() { mtlxBinary = orig }) + dir := t.TempDir() + spinloopPath := filepath.Join(dir, "Spinloop") + mustWrite(t, spinloopPath, "PROVIDER mtplx\nMODEL org/model\n") + + var err error + captureStdout(t, func() { err = cmdServe([]string{spinloopPath}) }) + if err == nil || !strings.Contains(err.Error(), "not found") { + t.Fatalf("expected a not-found error, got %v", err) + } + if !strings.Contains(err.Error(), "MTPLX") { + t.Errorf("hint should name MTPLX, got %v", err) + } +} + // TestCmdServe_UnsupportedProvider pins the behaviour change: serve used to run // llama-server whatever the PROVIDER said, which quietly served the wrong engine. func TestCmdServe_UnsupportedProvider(t *testing.T) { @@ -683,7 +884,9 @@ func TestCmdServe_UnsupportedProvider(t *testing.T) { if err == nil { t.Fatal("expected an error for a provider that is not a local engine") } - for _, want := range []string{"ollama", "llamacpp", "omlx"} { + // The list of servable engines is derived from the engine set, so the + // error names every one of them — including the newest. + for _, want := range []string{"ollama", "llamacpp", "omlx", "mtplx", "vllm"} { if !strings.Contains(err.Error(), want) { t.Errorf("error should mention %q, got %v", want, err) } diff --git a/docs/README.md b/docs/README.md index e57a743b..0ed09777 100644 --- a/docs/README.md +++ b/docs/README.md @@ -15,8 +15,8 @@ Four words carry the whole tool: Pi and lucinate are also supported. Chosen at runtime, so the same selection works for any of them. See [`spinloop harness`](commands/harness.md). - **Provider** — what `spinloop` can configure, from a built-in - catalogue: OpenRouter, AWS Bedrock, Ollama, llama.cpp, vLLM, oMLX (Apple - Silicon), or any OpenAI-compatible endpoint. See + catalogue: OpenRouter, AWS Bedrock, Ollama, llama.cpp, vLLM, oMLX and MTPLX + (both Apple Silicon), or any OpenAI-compatible endpoint. See [`spinloop list`](commands/list.md). - **Spinloop file** — a small, declarative file (like a `Dockerfile`, but for your agent's model) that captures one selection so you can commit it and @@ -83,7 +83,7 @@ including the `SPINLOOP_REMOTE_*` overrides: | `SPINLOOP_LOG_LEVEL` | How much `spinloop daemon`/`spinloop serve` record — `debug`, `info` (default), `warn`, `error` (`--log-level` beats it) | | *(named by `tokenEnv`)* | A [fleet](commands/fleet.md) node's bearer token — `fleet.yaml` names the variable, never the value | | `DEEPSEEK_API_KEY`, `OPENAI_API_KEY`, … | Provider API keys — `spinloop list` shows which each provider reads | -| `OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`, `OMLX_BASE_URL`, `VLLM_BASE_URL`, `OPENAI_BASE_URL` | Per-provider endpoint overrides | +| `OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`, `OMLX_BASE_URL`, `VLLM_BASE_URL`, `MTPLX_BASE_URL`, `OPENAI_BASE_URL` | Per-provider endpoint overrides | | `AWS_REGION` | Region for AWS Bedrock | Keys are looked up in a `.env` file **beside the `Spinloop` being applied** first @@ -93,5 +93,6 @@ it, the same way `PRESET` and `REMOTE` travel with a Spinloop. They are **never a reference the agent resolves when it runs, and `spinloop harness` passes the keys it can resolve to the agent it launches. If you start the agent yourself, set the variable in your own environment. Local providers on localhost (Ollama, -llama.cpp) need no key; Bedrock uses your AWS credentials. oMLX needs one only if -you enabled its API-key auth — set `OPENAI_API_KEY` before applying if you did. +llama.cpp) need no key; Bedrock uses your AWS credentials. oMLX and MTPLX need +one only if you enabled their API-key auth — set `OPENAI_API_KEY` before +applying if you did. diff --git a/docs/commands/remote.md b/docs/commands/remote.md index cc21ca91..2678d3a6 100644 --- a/docs/commands/remote.md +++ b/docs/commands/remote.md @@ -369,7 +369,9 @@ included) for every `kind: remote` node a fleet file names — or a chosen few - `deploy_url` is optional: a config written before `deploy` existed still works for `start`, `stop`, and `status`. - Only a self-hosted engine can be deployed (`llamacpp` or `vllm`). A hosted - provider has nothing to deploy. + provider has nothing to deploy. `mtplx` is a self-hosted engine too, but it is + Apple-Silicon-only and has no machine image, so it is not a cloud runner — it + serves locally and on a [fleet node](fleet.md), never on the cloud GPU. ## See also diff --git a/docs/commands/serve.md b/docs/commands/serve.md index ee7eb3b4..7b75c666 100644 --- a/docs/commands/serve.md +++ b/docs/commands/serve.md @@ -30,6 +30,7 @@ GPU: | `llamacpp` | `llama-server` | | `omlx` | [oMLX](https://omlx.ai) on Apple Silicon | | `vllm` | `vllm serve` (the model as its positional argument) | +| `mtplx` | [`mtplx serve`](https://mtplx.com) on Apple Silicon | Any other provider is an error: `serve` launches a self-hosted engine, and the rest of the catalogue names endpoints somebody else runs. @@ -49,6 +50,7 @@ engine serves it. `PARALLEL` sets the number of concurrent request slots, and | `llamacpp` | `--parallel n` | Scaled: `--ctx-size` becomes `context * n` | | `vllm` | `--max-num-seqs n` | Unscaled: `--max-model-len` is unaffected | | `omlx` | `--max-concurrent-requests n` | No context flag either way | +| `mtplx` | `--max-active-requests n` | Unscaled: `--context-window` is unaffected | The `llamacpp` scaling exists because llama.cpp's own `--ctx-size` is a total KV-cache budget it divides across `--parallel` slots — so without help, asking @@ -57,10 +59,12 @@ for `CONTEXT 128k` with two parallel slots would silently give each request `--ctx-size 256000 --parallel 2`, so each slot still gets the 128k the Spinloop asked for. -`vllm` and `omlx` need no such compensation: both share one dynamically-sized +`vllm` and `mtplx` need no such compensation: both share one dynamically-sized KV-cache pool across concurrent requests via continuous batching rather than -dividing a fixed budget per slot, so their own context settings are already a -per-request ceiling — `PARALLEL` only caps how many requests run at once. +dividing a fixed budget per slot, so their own context settings +(`--max-model-len`, `--context-window`) are already a per-request ceiling — +`PARALLEL` only caps how many requests run at once. `omlx` has no context flag +at all, so there is nothing to compensate there either. If `PARALLEL` is left out entirely, it changes nothing — no `--parallel`-family flag is added, and `CONTEXT` maps to the engine's context flag exactly as it @@ -201,6 +205,48 @@ it, unlike llama.cpp's. GPUs) are a different concept from `PARALLEL` and are not derived from it — set them by hand in a `PRESET`. +## MTPLX + +[MTPLX](https://mtplx.com) serves optimised models on Apple Silicon. Like +llama.cpp it is launched with one model, so a `MODEL` is required: + +```dockerfile +PROVIDER mtplx +MODEL Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed # an HF repo, or a local path +ALIAS qwen # mtplx serve --model-id +CONTEXT 128k # mtplx serve --context-window +PARALLEL 2 # mtplx serve --max-active-requests (see below) +BASEURL http://127.0.0.1:8000/v1 # mtplx serve --host/--port +``` + +- `MODEL` becomes `--model`, taken verbatim — an Hugging Face repo id or a local + path alike. `--download` is always passed, so a repo that is not already + local is fetched by the engine rather than failing the launch. +- `ALIAS` becomes `--model-id` — the name the model is served under. +- `CONTEXT` becomes `--context-window` — a per-request ceiling, never scaled by + `PARALLEL`: see [Parallelism](#parallelism) above. +- `PARALLEL` becomes `--max-active-requests` — an admission cap on how many + requests run at once. It never selects the engine's scheduling mode. +- `BASEURL` sets the bind address. With none, no bind flag is emitted and + MTPLX's own defaults stand. + +The scheduling mode (`--scheduler-mode`: `serial`, `parallel`, `concurrent`) is +per-deployment tuning, not a Spinloop field — set it in a `PRESET`, written in +MTPLX's own long-form flags. `serve` passes every other preset key through +unchanged, so a preset is portable only to MTPLX, as with every engine. + +`serve` never passes `--api-key`, for the same reason as [oMLX](#omlx): it +prints the command it runs, and a key on the line would be in your screen and +the process table. A supervised engine (`--api` or the daemon) is gated with a +key file the daemon writes instead. Because the `mtplx` provider is +`apiKeyOptional`, `spinloop add`/`apply` only writes the key reference when +`OPENAI_API_KEY` is set at apply time. + +### Finding the binary + +`serve` looks for `mtplx` on your `PATH`. Install it from +[mtplx.com](https://mtplx.com) if it is not there. + ## The control API (`--api`) and `spinloop daemon` `serve` is strictly foreground: it runs the engine in front of you until one diff --git a/docs/env-vars.md b/docs/env-vars.md index 8d054298..50d98041 100644 --- a/docs/env-vars.md +++ b/docs/env-vars.md @@ -45,7 +45,7 @@ naming one) rather than environment variables alone. | `XDG_CONFIG_HOME` | Base for spinloop's config dir (`$XDG_CONFIG_HOME/spinloop`) when `SPINLOOP_CONFIG_DIR` is unset. | | `AWS_REGION` | AWS region for the remote control calls when the remote config names none. | | `HF_TOKEN` | Hugging Face token, used only to seed gated model weights during `spinloop remote deploy`. | -| `OPENAI_API_KEY` | The key spinloop resolves for OpenAI-compatible and oMLX providers (from the environment or the adjacent `.env`). | +| `OPENAI_API_KEY` | The key spinloop resolves for OpenAI-compatible, oMLX and MTPLX providers (from the environment or the adjacent `.env`). | Each provider in the catalogue also names its own key variable (and sometimes a base-URL or region variable); `spinloop list` shows the provider details, and the diff --git a/docs/openapi.yaml b/docs/openapi.yaml index 4f6ac26a..180c13f8 100644 --- a/docs/openapi.yaml +++ b/docs/openapi.yaml @@ -294,7 +294,7 @@ components: runner: type: string description: The engine being served, when known. - examples: [llamacpp, vllm] + examples: [llamacpp, omlx, vllm, mtplx] model: type: string description: What it is serving, when known. @@ -489,7 +489,10 @@ components: properties: runner: type: string - enum: [llamacpp, vllm] + # The engines a deploy config can name. mtplx appears because a fleet + # node can be woken with it; it is not a cloud runner (no machine + # image). omlx is not a wakeable runner yet. + enum: [llamacpp, vllm, mtplx] modelId: type: string description: The weights to serve — a Hugging Face repo, or a local path on the instance. diff --git a/docs/spinloop-file.md b/docs/spinloop-file.md index 3bdafd91..a92fbec8 100644 --- a/docs/spinloop-file.md +++ b/docs/spinloop-file.md @@ -195,11 +195,12 @@ Rules: budget it divides across its `--parallel` slots, a `llamacpp` Spinloop with both `CONTEXT` and `PARALLEL` set gets a `--ctx-size` scaled by the slot count so each slot still gets what `CONTEXT` promised (`CONTEXT 128k` + - `PARALLEL 2` → `--ctx-size 256000 --parallel 2`). `vllm` and `omlx` have no - such coupling — `PARALLEL` becomes `--max-num-seqs`/`--max-concurrent-requests` - respectively, and `CONTEXT` is never scaled by it. See - [`spinloop serve`](commands/serve.md#parallelism) for the full per-engine - mapping. + `PARALLEL 2` → `--ctx-size 256000 --parallel 2`). `vllm`, `mtplx`, and `omlx` + have no such coupling — `PARALLEL` becomes `--max-num-seqs`/ + `--max-active-requests`/`--max-concurrent-requests` respectively, and + `CONTEXT` is never scaled by it. See + [`spinloop serve`](commands/serve.md#parallelism) for the full per-engine + mapping. - `BASEURL` overrides the provider's API base URL — handy for a gateway or a llama.cpp server on a non-default port. `URL`, `BASE-URL`, and `BASE_URL` are accepted as aliases. diff --git a/examples/mtplx/qwen3.8-27b/README.md b/examples/mtplx/qwen3.8-27b/README.md new file mode 100644 index 00000000..f629f089 --- /dev/null +++ b/examples/mtplx/qwen3.8-27b/README.md @@ -0,0 +1,119 @@ +# Qwen3.8-27B on MTPLX (Apple Silicon) + +Run an MTPLX-optimised build of Qwen3.8-27B on your Mac with +[MTPLX](https://mtplx.com), then point opencode at it with the +[`Spinloop`](Spinloop) in this directory. + +MTPLX is a single-model server: you launch it with one model and it serves that +model over an OpenAI-compatible API. That makes it the closest of the local +engines to +[llama.cpp](../../llamacpp/qwen3.8-27b/README.md), but tuned for Apple Silicon +unified memory. + +## Prerequisites + +- An **Apple Silicon** Mac (M1 or later). MTPLX is Apple Silicon only — there is + no Intel or Linux build, and **no machine image**, so + [`spinloop remote`](../../../docs/commands/remote.md) cannot deploy it. It + serves locally, or on a + [fleet node](../../../docs/commands/fleet.md) you run yourself. +- [MTPLX](https://mtplx.com), installed so that `mtplx` is on your `PATH`. +- Enough unified memory for the weights. + +## 1. Start MTPLX + +The raw command, for reference: + +```sh +mtplx serve \ + --model Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed \ + --model-id qwen3.8-27b \ + --context-window 32768 \ + --max-active-requests 4 \ + --scheduler-mode parallel \ + --download \ + --host 127.0.0.1 --port 8000 +``` + +What the flags do: + +- `--model` — the Hugging Face repo to serve, or a local path. Taken verbatim. +- `--download` — fetch the repo if it is not already local, instead of failing + the launch. `spinloop serve` always passes this. +- `--model-id` — the name the model is served under (the `ALIAS`). +- `--context-window` — the context a single request gets. Never scaled by + `PARALLEL`. +- `--max-active-requests` — an admission cap on how many requests run at once. +- `--scheduler-mode` — `serial`, `parallel`, or `concurrent`. This is + per-deployment tuning, not a Spinloop field, so it lives in the + [`preset.ini`](preset.ini). +- `--host`/`--port` — the OpenAI-compatible API is served at + `http://127.0.0.1:8000/v1`. + +Rather than remember those flags, this directory keeps them in a +[`preset.ini`](preset.ini) and lets `spinloop` build and run the command: + +```sh +spinloop serve # from this directory; reads ./Spinloop and its PRESET +spinloop serve --dry-run # print the `mtplx serve` command without running it +``` + +The preset is read in MTPLX's own long-form vocabulary — its keys pass through +unchanged — so it is not interchangeable with a llama.cpp preset. + +### Check it's up + +```sh +curl http://127.0.0.1:8000/v1/models +curl http://127.0.0.1:8000/health +``` + +`/health` answers once the OpenAI server is up, which may be before the weights +finish loading. + +## 2. Point opencode at it + +MTPLX speaks the OpenAI-compatible API, which is what the `mtplx` provider +targets (default base URL `http://localhost:8000/v1`, overridable with +`MTPLX_BASE_URL`). Apply the [`Spinloop`](Spinloop) in this directory: + +```sh +spinloop apply examples/mtplx/qwen3.8-27b/Spinloop +# or, from this directory: +spinloop apply +``` + +The Spinloop is: + +```dockerfile +PROVIDER mtplx +ALIAS qwen3.8-27b +CONTEXT 32768 +PARALLEL 4 +PRESET ./preset.ini +``` + +`CONTEXT` maps to `--context-window` and `PARALLEL` to `--max-active-requests`, +each in MTPLX's own vocabulary. Running on a different host or port? Add a +`BASEURL` line (the file ships one commented out) — it sets both what opencode +calls and what `serve` binds to. + +## A note on API keys + +By default `spinloop` writes no key for a localhost MTPLX: the `mtplx` provider +is marked `apiKeyOptional`, so a local endpoint with no `OPENAI_API_KEY` set +gets no `apiKey` field at all. If you put a key on the engine, set +`OPENAI_API_KEY` **before** you apply the Spinloop, and `spinloop` writes an +`{env:OPENAI_API_KEY}` reference (never the secret). + +`spinloop serve` never passes `--api-key`: it prints the command it runs, so a +key there would land on your screen and in the process table. A supervised +engine (`serve --api` or `spinloop daemon`) is gated with a key file the daemon +writes instead. + +## See also + +- [`spinloop serve`](../../../docs/commands/serve.md) — the full command reference +- [The `Spinloop` file](../../../docs/spinloop-file.md) — full syntax +- The same model on llama.cpp: + [`examples/llamacpp/qwen3.8-27b`](../../llamacpp/qwen3.8-27b/README.md) diff --git a/examples/mtplx/qwen3.8-27b/Spinloop b/examples/mtplx/qwen3.8-27b/Spinloop new file mode 100644 index 00000000..81358370 --- /dev/null +++ b/examples/mtplx/qwen3.8-27b/Spinloop @@ -0,0 +1,10 @@ +# Qwen3.8-27B served locally by MTPLX on Apple Silicon. +# ALIAS names the model in opencode and selects the preset section below. +PROVIDER mtplx +ALIAS qwen3.8-27b +CONTEXT 32768 # --context-window; raise it on a bigger box +PARALLEL 4 # --max-active-requests; the scheduler mode is in the preset +PRESET ./preset.ini # `spinloop serve` launches `mtplx serve` from this +# BASEURL http://127.0.0.1:8000/v1 # uncomment for a non-default host/port +# REMOTE qwen3.8-27b # not applicable: mtplx is Apple-Silicon-only and has +# # no machine image, so `spinloop remote` cannot deploy it diff --git a/examples/mtplx/qwen3.8-27b/preset.ini b/examples/mtplx/qwen3.8-27b/preset.ini new file mode 100644 index 00000000..b5b5ab91 --- /dev/null +++ b/examples/mtplx/qwen3.8-27b/preset.ini @@ -0,0 +1,18 @@ +# An MTPLX preset for Qwen3.8-27B, mirroring the `mtplx serve` command in +# this directory's README. `spinloop serve` turns it into that command and +# runs it. +# +# These are `mtplx serve` flags with their leading dashes stripped, in MTPLX's +# own long-form vocabulary. They are NOT llama.cpp's: each engine's preset is +# read in its own vocabulary, so this file is not interchangeable with the +# ones under examples/llamacpp/. + +[*] +host = 127.0.0.1 +port = 8000 + +[qwen3.8-27b] # the Spinloop's ALIAS selects this section +model = Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed +context-window = 32768 # keep in step with the Spinloop's CONTEXT +max-active-requests = 4 # the Spinloop's PARALLEL, when it states one +scheduler-mode = parallel # serial | parallel | concurrent — per-deployment diff --git a/internal/catalog/catalog_test.go b/internal/catalog/catalog_test.go index be7daa09..1b6bb7f8 100644 --- a/internal/catalog/catalog_test.go +++ b/internal/catalog/catalog_test.go @@ -648,6 +648,74 @@ func TestBuildProviderBlock_LlamacppReferencesKeyForRemote(t *testing.T) { } } +// MTPLX, like oMLX and llama.cpp, is an optional-key local OpenAI-compatible +// engine: a local server gets no key, a set key is referenced (never written +// as a literal), and Pi/lucinate get their usual keyless and resolved forms. +func TestMtplxProviderKeyHandling(t *testing.T) { + cat, _ := Load() + p, ok := cat.Providers["mtplx"] + if !ok { + t.Fatal("catalogue has no mtplx provider") + } + if !p.APIKeyOptional { + t.Fatal("mtplx key should be optional") + } + + withKey := func(name string) string { + if name == "OPENAI_API_KEY" { + return "sk-mtplx" + } + return "" + } + + // opencode, keyless local: no apiKey option at all. + block, _, err := BuildProviderBlock("mtplx", p, "local-model", "", noEnv) + if err != nil { + t.Fatalf("BuildProviderBlock: %v", err) + } + if _, ok := block["options"].(map[string]any)["apiKey"]; ok { + t.Error("a keyless local mtplx server should get no apiKey option") + } + + // opencode, key set: an env reference, never the literal secret. + block, _, err = BuildProviderBlock("mtplx", p, "local-model", "", withKey) + if err != nil { + t.Fatalf("BuildProviderBlock: %v", err) + } + opts := block["options"].(map[string]any) + if opts["apiKey"] != "{env:OPENAI_API_KEY}" { + t.Errorf("apiKey = %v, want the env reference", opts["apiKey"]) + } + if opts["apiKey"] == "sk-mtplx" { + t.Error("the resolved key must never be written into the block") + } + + // Pi: the placeholder when keyless, the $VAR reference when set. + prov, _, err := BuildPiProvider("mtplx", p, "local-model", "", noEnv) + if err != nil { + t.Fatalf("BuildPiProvider: %v", err) + } + if prov.APIKey != piPlaceholderAPIKey { + t.Errorf("Pi apiKey (keyless) = %q, want the placeholder", prov.APIKey) + } + prov, _, err = BuildPiProvider("mtplx", p, "local-model", "", withKey) + if err != nil { + t.Fatalf("BuildPiProvider: %v", err) + } + if prov.APIKey != "$OPENAI_API_KEY" { + t.Errorf("Pi apiKey (set) = %q, want the $VAR reference", prov.APIKey) + } + + // lucinate: accepted, resolving a concrete endpoint. + conn, _, err := BuildLucinateConnection("mtplx", p, "local-model", "", noEnv) + if err != nil { + t.Fatalf("BuildLucinateConnection: %v", err) + } + if conn.BaseURL == "" { + t.Error("lucinate connection should carry a resolved base URL") + } +} + func TestIsLocalEndpoint(t *testing.T) { for _, u := range []string{ "", "http://localhost:8080/v1", "http://127.0.0.1:8080/v1", diff --git a/internal/catalog/providers.yaml b/internal/catalog/providers.yaml index b62f2410..41e5f5dc 100644 --- a/internal/catalog/providers.yaml +++ b/internal/catalog/providers.yaml @@ -143,6 +143,24 @@ providers: api: openai-completions lucinate: {} + mtplx: + description: MTPLX server on Apple Silicon (OpenAI-compatible API) + name: MTPLX + npm: "@ai-sdk/openai-compatible" + # Like oMLX, a local MTPLX server usually needs no key and none is injected + # when the var is unset; MTPLX can be started with an API key, so one is + # picked up when there is one (e.g. a Mac on the LAN named by + # MTPLX_BASE_URL). + apiKeyEnv: OPENAI_API_KEY + apiKeyOptional: true + options: + baseURL: http://localhost:8000/v1 + optionsFromEnv: + baseURL: MTPLX_BASE_URL + pi: + api: openai-completions + lucinate: {} + vllm: description: vLLM server (OpenAI-compatible API) name: vLLM diff --git a/internal/daemon/activity.go b/internal/daemon/activity.go index ad5a5a6a..84438aef 100644 --- a/internal/daemon/activity.go +++ b/internal/daemon/activity.go @@ -130,7 +130,9 @@ func (d *Daemon) sampleOnce(ctx context.Context) { d.mu.Lock() scrape := d.scrape d.mu.Unlock() - if scrape.BaseURL == "" { + // A target with an address but no dialect has no /metrics to parse — the + // address exists so the readiness check can probe /health, not to scrape. + if scrape.BaseURL == "" || scrape.Engine == "" { return } tokens, err := metrics.ScrapeTokenStats(ctx, scrape) @@ -143,20 +145,24 @@ func (d *Daemon) sampleOnce(ctx context.Context) { // checkReadyOnce takes one reading of whether the running engine can serve // requests, for a runner with a known health-check convention. It is a no-op -// — recording nothing — when the engine is not running, when no scrape -// target is known, or when the runner is not in readinessCheckedRunners, so -// an unchecked runner's readiness field stays absent rather than reporting a -// guess. +// — recording nothing — when the engine is not running, when no address is +// known, or when the runner is not in readinessCheckedRunners, so an +// unchecked runner's readiness field stays absent rather than reporting a +// guess. The convention is keyed on the runner, not the metrics dialect: an +// engine with no /metrics to scrape can still answer /health at its own +// address, and that is what this check probes. func (d *Daemon) checkReadyOnce(ctx context.Context) { if state, _, _ := d.Sup.Status(); state != StateRunning { return } - // Copy the target under the lock and release before the HTTP call, as - // sampleOnce does — a health check must never hold the daemon's mutex. + // Copy the address and runner under the lock and release before the HTTP + // call, as sampleOnce does — a health check must never hold the daemon's + // mutex. d.mu.Lock() scrape := d.scrape + runner := d.runner d.mu.Unlock() - if scrape.BaseURL == "" || !readinessCheckedRunners[scrape.Engine] { + if scrape.BaseURL == "" || !readinessCheckedRunners[runner] { return } d.ready.record(metrics.CheckEngineReady(ctx, scrape)) diff --git a/internal/daemon/activity_test.go b/internal/daemon/activity_test.go index 77f2fc63..678405f5 100644 --- a/internal/daemon/activity_test.go +++ b/internal/daemon/activity_test.go @@ -278,9 +278,10 @@ while true; do sleep 0.05; done`) } } -// TestSampleOnceNoTarget covers the two quiet paths: an engine with no metrics -// endpoint is skipped, and an unreachable one records nothing rather than -// being read as either active or idle. +// TestSampleOnceNoTarget covers the quiet paths: an engine with no metrics +// endpoint — or an address with no dialect to parse — is skipped, and an +// unreachable one records nothing rather than being read as either active or +// idle. func TestSampleOnceNoTarget(t *testing.T) { d := testDaemon(t, `trap 'exit 0' TERM while true; do sleep 0.05; done`) @@ -302,6 +303,14 @@ while true; do sleep 0.05; done`) t.Error("sampled with no scrape target") } + // An address with no dialect has no /metrics to parse — the address exists + // for the readiness check, not for the sampler. + d.SetScrape(metrics.ScrapeTarget{BaseURL: "http://127.0.0.1:1", Engine: ""}) + d.sampleOnce(context.Background()) + if _, ok := d.act.snapshot(); ok { + t.Error("sampled an engine with no metrics dialect") + } + // A target that does not answer: a non-observation, not activity. d.SetScrape(metrics.ScrapeTarget{BaseURL: "http://127.0.0.1:1", Engine: "llamacpp"}) d.sampleOnce(context.Background()) diff --git a/internal/daemon/daemon.go b/internal/daemon/daemon.go index 3ad06aad..817d0329 100644 --- a/internal/daemon/daemon.go +++ b/internal/daemon/daemon.go @@ -86,9 +86,11 @@ func (d *Daemon) now() time.Time { return time.Now() } -// SetScrape records where the running engine's own /metrics lives; an empty -// BaseURL means no engine scrape (an engine with no metrics endpoint). It is -// set alongside each start, so it always describes the engine that runs. +// SetScrape records where the running engine's own endpoint lives; an empty +// BaseURL means the address could not be determined. The Engine names the +// metrics dialect to scrape — empty for an engine with no /metrics — while +// the BaseURL is still used to probe /health for readiness. It is set +// alongside each start, so it always describes the engine that runs. func (d *Daemon) SetScrape(target metrics.ScrapeTarget) { d.mu.Lock() d.scrape = target diff --git a/internal/daemon/readiness.go b/internal/daemon/readiness.go index 8c04d0ab..b2c6c5f9 100644 --- a/internal/daemon/readiness.go +++ b/internal/daemon/readiness.go @@ -10,6 +10,7 @@ import "sync" var readinessCheckedRunners = map[string]bool{ "llamacpp": true, "vllm": true, + "mtplx": true, } // readiness is the daemon's record of whether the running engine last diff --git a/internal/daemon/readiness_test.go b/internal/daemon/readiness_test.go index 0d140277..50cd45a1 100644 --- a/internal/daemon/readiness_test.go +++ b/internal/daemon/readiness_test.go @@ -127,6 +127,32 @@ func TestCheckReadyOnceUnknownRunnerSkipped(t *testing.T) { } } +// TestCheckReadyOnceNoMetricsDialectStillChecked covers mtplx: the engine has +// no /metrics to scrape, so its scrape target carries an address but no +// dialect, yet it still answers /health. The readiness check must run on the +// address alone — keying it on the runner, not the dialect — so a node that +// cannot be scraped can still be read as ready or not. +func TestCheckReadyOnceNoMetricsDialectStillChecked(t *testing.T) { + health := &fakeHealth{status: http.StatusOK} + srv := httptest.NewServer(health) + defer srv.Close() + + d := startRunningDaemon(t, "mtplx") + // No dialect: this is exactly what scrapeTargetFor yields for mtplx. + d.SetScrape(metrics.ScrapeTarget{BaseURL: srv.URL, Engine: ""}) + + d.checkReadyOnce(context.Background()) + if got := d.readinessField(); got != "ready" { + t.Fatalf("readinessField for a dialect-less engine = %q, want ready", got) + } + if got := d.Status().Ready; got != "ready" { + t.Errorf("Status().Ready for a dialect-less engine = %q, want ready", got) + } + if got := d.Metrics(context.Background()).Ready; got != "ready" { + t.Errorf("Metrics().Ready for a dialect-less engine = %q, want ready", got) + } +} + // TestReadinessAbsentWhenNotRunning covers idle, stopped, and crashed alike: // Status and Metrics only consult readiness inside their running branch, so // a stale reading never leaks out once the engine is not running. diff --git a/internal/preset/preset.go b/internal/preset/preset.go index 5b4ad64b..49aeea21 100644 --- a/internal/preset/preset.go +++ b/internal/preset/preset.go @@ -191,6 +191,11 @@ var VLLM = Dialect{} // aliasing is applied and keys pass through as written. var OMLX = Dialect{} +// MTPLX is the dialect `mtplx serve` speaks. Every flag is long form +// (`--model`, `--context-window`, `--max-active-requests`), so no aliasing is +// applied and keys pass through as written. +var MTPLX = Dialect{} + // Flags merges ordered layers of params into flag tokens. Later layers override // earlier ones by canonical flag name, in place, so the first layer fixes the // order: pass globals, then the section, then any overrides. It does not diff --git a/openspec/changes/archive/2026-09-04-add-mtplx-engine/design.md b/openspec/changes/archive/2026-09-04-add-mtplx-engine/design.md new file mode 100644 index 00000000..d9e47019 --- /dev/null +++ b/openspec/changes/archive/2026-09-04-add-mtplx-engine/design.md @@ -0,0 +1,114 @@ +# Design + +## The engine table entry + +`engineFor` gains a `mtplx` case returning the MTPLX serve params: + +| Spinloop field | flag | notes | +| -------------- | --------------------- | ------------------------------------------------------------ | +| MODEL | `--model` | an HF repo id or a local path, verbatim | +| ALIAS | `--model-id` | omitted when unset; the engine derives a name otherwise | +| CONTEXT | `--context-window` | a token count, parsed as today | +| PARALLEL | `--max-active-requests` | an admission cap; see the Parallelism delta | +| BASEURL | `--host` / `--port` | omitted entirely when unset, so MTPLX's own defaults stand | + +- `needsModel` is true. spinloop forbids a silent default model, and MTPLX + has one; the Spinloop names what it serves. +- `--download` is always passed. MTPLX's auto-fetch applies to its own + optimised packs and is a no-op when the model is already present, which is + the closest analogue to `llama-server -hf`. Other weight formats are + obtained with `mtplx pull`, which stays outside this capability. +- A key, when set, renders as `--api-key-file `: the daemon writes the + key to a `0600` file and the secret never appears on the command line, the + same discipline as every other engine. +- The binary is expected on the `PATH`; the brew, pip, and app installs all + put it there, so there is no conventional install location to fall back to. +- The preset dialect is the zero value: every MTPLX preset key is a long-form + flag, so keys pass through unchanged and the override-by-canonical-name + rule needs no alias table. + +## Readiness + +`readinessCheckedRunners` gains `mtplx`. MTPLX's `/health` answers +`200 {"ok":true}` once its OpenAI server is up, which may be before the +weights finish loading. That is fine: routing treats `running` as "a process +exists" and launches only once the engine endpoint actually answers, so a +still-loading node is waited for — the existing "A started engine that is not +yet loaded is waited for" behaviour, unchanged. + +The map alone is not enough, because the readiness check needs the engine's +*address*, and that was only being carried on the metrics scrape target — +which mtplx has none of, since it has no /metrics dialect. So the address is +decoupled from the dialect: + +- `scrapeTargetFor` always resolves the address (the engine's own `--host`/ + `--port`, else BASEURL, else the engine default) even when the engine has no + metrics dialect; the target's dialect field stays empty in that case. +- The activity sampler skips a target that has an address but no dialect — + there is no /metrics to parse, so an engine that cannot be scraped still + costs nothing. +- The readiness check is keyed on the **runner**, not the dialect. That is + what the daemon-api spec's "the runner has a known health-check convention" + actually means, and it is what lets a dialect-less engine like mtplx be + probed at `/health` on its own address. omlx stays unchecked: it is not in + the map, so its readiness field remains absent. + +## Metrics: none + +MTPLX's `/metrics` is a JSON ring of per-request envelopes, not a Prometheus +text endpoint with cumulative counters. There is no dialect to write, so +`mtplx` gets the omlx treatment: no scrape target, no token stats, the +activity sampler samples nothing, and a fleet node's "last active" figure is +omitted rather than implied. If MTPLX ever grows cumulative counters, that is +a separate change with its own dialect. + +## The runner-aware Spinloop-to-deploy-config path + +`deployConfig` (cmd/spinloop/remote.go) is shared by the cloud and node +paths, and today hardcodes llama.cpp-shaped assumptions. The change makes the +node path runner-aware and leaves the cloud path byte-for-byte as it is: + +- **The gate moves to the target.** The cloud keeps its runners — `llamacpp` + and `vllm` — and its error message. The node path accepts `llamacpp`, + `vllm`, and `mtplx`. MTPLX is Apple-Silicon-only and has no machine image, + so it never becomes a cloud runner. Every consumer of the node path — a + routed wake, `spinloop fleet start` (which the fleet-client spec says uses + the same node-owned derivation), and the `fleet route` dry-run — gains + mtplx together, so no caller needs its own change. +- **Preset fallback keys become per-runner.** The model falls back to the + preset's `hf` key for `llamacpp` and `vllm` (unchanged) and to `model` for + `mtplx`. The context falls back to `ctx-size` for `llamacpp` and `vllm` + (unchanged) and to `context-window` for `mtplx`. +- **The local-path check follows the weights.** A local model path is + refused only where the destination fetches the weights itself (the cloud): + it cannot ship a file. A node has the file the Spinloop points at, so the + node path carries the path as the model to load. Today the check is + unconditional and fires even on the node path, which blocks llamacpp and + vllm wakes with local weights as well; moving it unblocks those too. +- **Quant splitting stays as it is.** `splitModelQuant` is a no-op for + mtplx: an HF repo id and a local path contain no colon, and a colon, if one + ever appeared, splits and rejoins at the same place when the daemon builds + the command, so the reference round-trips exactly. +- **Owned preset keys.** `model` and `api-key`/`api-key-file` are already in + the owned set. `model-id`, `context-window`, and `max-active-requests` + join it, so a preset's raw values do not double-define the computed + flags. `host` and `port` stay node-owned and preserved, as now. +- **`parallelPresetKey` gains its `mtplx` case** — `max-active-requests` — + which the node path makes reachable. Its `omlx` case stays unreachable + until #159. + +The daemon side needs nothing new: `validateDeployConfig` resolves the runner +through `engineFor`, which now knows `mtplx`, and `BuildArgv` renders a +deploy config through the same engine table as `serve`. + +## Non-goals + +- **omlx fleet wake.** The runner-aware path makes it a small follow-up + (#159); keeping it out keeps this change's deltas reviewable. +- **The scheduling mode as a Spinloop field.** `--scheduler-mode` (serial / + parallel / concurrent) is per-deployment tuning that belongs in a preset, + the same way `--speculative` and friends do elsewhere. `PARALLEL` only + admits requests; how they execute is the preset's call. +- **The MTPLX app's own server.** If the app is already serving on the port, + `mtplx serve` exits naming the occupant; there is no attach mode, and + spinloop does not try to adopt a foreign process. diff --git a/openspec/changes/archive/2026-09-04-add-mtplx-engine/proposal.md b/openspec/changes/archive/2026-09-04-add-mtplx-engine/proposal.md new file mode 100644 index 00000000..28a8bd37 --- /dev/null +++ b/openspec/changes/archive/2026-09-04-add-mtplx-engine/proposal.md @@ -0,0 +1,79 @@ +# Add mtplx engine support + +## Why + +MTPLX is an MLX-based inference server for Apple Silicon with native MTP +speculative decoding — for the optimised models it ships, it decodes several +times faster than other local engines, which is why Macs running coding +workloads increasingly use it. It is OpenAI-compatible and binds the model it +serves at start, which makes it a natural fit for spinloop: servable, +supervisable, and routable. Today a `PROVIDER mtplx` Spinloop is none of +those — the engine table knows only `llamacpp`, `omlx`, and `vllm`, the +catalogue has no `mtplx` entry, and the fleet cannot wake a node for it. + +## What Changes + +- `mtplx` becomes a servable engine. `PROVIDER mtplx` runs `mtplx serve`, + with the model as `--model`, the served name as `--model-id`, the context + window as `--context-window`, and a `--download` flag so the engine fetches + a model it does not have itself, the way `llama-server -hf` does. MTPLX + preset keys are all long-form, so presets pass through unchanged. +- For `mtplx`, `PARALLEL n` renders as `--max-active-requests n`, MTPLX's + admission cap on concurrent requests. The scheduling mode — how admitted + requests execute — stays a preset concern: `PARALLEL` never selects it. +- The fleet can wake a node for `mtplx`. The Spinloop-to-deploy-config + derivation becomes runner-aware: the node path accepts `mtplx` alongside + `llamacpp` and `vllm`, per-engine preset fallback keys replace the + llama.cpp-only ones, and a `MODEL` naming a file on the node's own disk is + a valid wake — today the local-path check fires even on the node path, + which also blocks llamacpp and vllm wakes with local weights. Cloud deploy + is unchanged: MTPLX is Apple-Silicon-only and has no machine image, so + `remote deploy` still refuses it. +- Readiness: the daemon gains a `/health` convention for `mtplx`, so a + started or woken node reports readiness the way llamacpp and vllm do. +- The catalogue gains an `mtplx` provider: an OpenAI-compatible endpoint + defaulting to `http://localhost:8000/v1`, overridable with + `MTPLX_BASE_URL`, optionally authenticated, Pi-capable, and + lucinate-capable. +- MTPLX exposes no Prometheus metrics — its `/metrics` is a JSON ring of + per-request envelopes — so it gets the omlx treatment: no scrape dialect, + no token stats, nothing sampled. +- Docs and an `examples/mtplx/` guide. + +## Capabilities + +### New Capabilities + +(none — this change modifies existing capabilities only) + +### Modified Capabilities + +- `local-serving`: "Choosing the engine" — `mtplx` SHALL run `mtplx serve` + with the model as `--model`, the served name as `--model-id`, the context + window as `--context-window`, and `--download` so a missing model is + fetched by the engine. "Parallelism" — for `mtplx`, `PARALLEL n` SHALL + render as `--max-active-requests n`, an admission cap that does not scale + the context; `PARALLEL` never selects MTPLX's scheduling mode. +- `fleet-routing`: "Waking a node" — a node may be woken for an engine that + binds its model at launch (`llamacpp`, `vllm`, `mtplx`), and a `MODEL` + naming a file on the node's own disk is a valid wake for it. +- `provider-catalog`: a new "MTPLX local provider" requirement — the + catalogue SHALL include an `mtplx` provider with the plumbing above. + +## Impact + +- Go client: `cmd/spinloop/serve.go` (the engine table's `mtplx` entry), + `cmd/spinloop/remote.go` (the runner-aware derivation: the node gate, + per-engine preset fallback keys, the local-path check moving to the + cloud-only path, the owned preset keys), `internal/daemon/readiness.go` + (the `/health` convention), `internal/catalog/providers.yaml` (the + `mtplx` entry). No `internal/preset` change — MTPLX's dialect is the zero + value. +- Non-goals: metrics (no Prometheus dialect), cloud deploy (no machine + image), and omlx fleet wake (tracked separately, #159). +- Docs: `docs/commands/serve.md`, `docs/spinloop-file.md`, + `docs/commands/remote.md`, `docs/openapi.yaml` (the runner examples), + `examples/mtplx/`. +- Tests: the Go suite (the engine table, preset rendering, readiness, + per-runner deploy-config derivation, the catalogue entry). Coverage stays + at 80% or above. diff --git a/openspec/changes/archive/2026-09-04-add-mtplx-engine/specs/fleet-routing/spec.md b/openspec/changes/archive/2026-09-04-add-mtplx-engine/specs/fleet-routing/spec.md new file mode 100644 index 00000000..b1fd8385 --- /dev/null +++ b/openspec/changes/archive/2026-09-04-add-mtplx-engine/specs/fleet-routing/spec.md @@ -0,0 +1,92 @@ +## MODIFIED Requirements + +### Requirement: Waking a node + +When no running node is serving what is wanted, routing SHALL wake one: it SHALL +choose a node that is not running, push what the Spinloop asks for as that node's +deploy config, start it through the daemon's start endpoint, and wait before +launching the agent — not merely until the node reports `running`, which says +only that a process exists, but until its engine endpoint answers. A node whose +stored config already matches the wanted model SHALL be preferred, since it has +the weights. + +The pushed config is the node-side counterpart of what `spinloop serve` would run +for that Spinloop, translated per engine. A node may be woken for an engine that +binds its model at launch — `llamacpp`, `vllm`, and `mtplx` — and a `MODEL` that +names a file on the node's own disk is a valid wake for it: the node has the +file, and only a destination that fetches its weights itself refuses a local +path. + +A node that refuses the config — a runner or model it cannot serve — SHALL NOT +fail the launch while other candidates remain: the next candidate SHALL be +tried, and the refusals SHALL be reported when none succeeds. + +Two clients may wake the same node at once. A start refused because an engine is +already running SHALL NOT fail the launch: the node's state SHALL be re-read, +and a node now serving what was wanted SHALL be used. Losing that race is +another route to the same place, not an error. + +The wait SHALL be bounded by a timeout and SHALL report what it is waiting for, +because a cold node loads weights before it answers. Exceeding the timeout SHALL +fail naming the node and the endpoint that did not come up; the started engine +SHALL be left running rather than stopped, so a slow load is not thrown away. + +`--no-wake` SHALL turn waking off: with no running node serving what is wanted +the command SHALL then fail, listing the nodes and their states and naming the +command that would start one. + +#### Scenario: An idle node is woken and used + +- **WHEN** a fleet-routed launch finds no node serving the wanted model and one + node is idle and able to serve it +- **THEN** that node is given the Spinloop's model as its deploy config, started, + and the agent launches against it once its engine answers + +#### Scenario: A node is woken for a Mac-only engine + +- **WHEN** a fleet-routed launch finds no node serving the wanted model, and an + idle node's daemon can run MTPLX +- **THEN** that node is woken with a config that runs the wanted model under + `mtplx serve`, and the agent launches against it once its engine answers + +#### Scenario: A local model path wakes the node that has it + +- **WHEN** the Spinloop's `MODEL` names a file on the woken node's disk +- **THEN** the wake carries that path as the model to load, rather than + refusing it as a local file + +#### Scenario: A started engine that is not yet loaded is waited for + +- **WHEN** a woken node reports `running` while its engine is still loading + weights and not yet answering +- **THEN** the launch waits for the engine to answer rather than launching the + agent against an endpoint that refuses connections + +#### Scenario: A node that cannot serve the model is passed over + +- **WHEN** the first idle candidate rejects the pushed config as unservable and + a second idle node accepts it +- **THEN** the second node is started and used + +#### Scenario: No node can serve it + +- **WHEN** every idle node rejects the config +- **THEN** the command fails, naming each node and the reason it refused + +#### Scenario: Losing the race to another client + +- **WHEN** a start is refused because another client woke the same node first, + and that node is now serving the wanted model +- **THEN** the launch uses that node rather than failing + +#### Scenario: A node that never comes up + +- **WHEN** a woken node does not report running within the timeout +- **THEN** the command fails naming the node, and the engine it started is left + running rather than stopped + +#### Scenario: Waking is refused + +- **WHEN** `--no-wake` is passed and no node is serving the wanted model +- **THEN** the command fails, listing the nodes with their states and naming the + `spinloop fleet start` command that would start one, and nothing is started diff --git a/openspec/changes/archive/2026-09-04-add-mtplx-engine/specs/local-serving/spec.md b/openspec/changes/archive/2026-09-04-add-mtplx-engine/specs/local-serving/spec.md new file mode 100644 index 00000000..e665371c --- /dev/null +++ b/openspec/changes/archive/2026-09-04-add-mtplx-engine/specs/local-serving/spec.md @@ -0,0 +1,158 @@ +## MODIFIED Requirements + +### Requirement: Choosing the engine + +`spinloop serve` SHALL launch the inference engine the Spinloop's `PROVIDER` names, +the local counterpart of the runner `spinloop remote deploy` selects from the same +instruction. `llamacpp` SHALL run `llama-server`; `omlx` SHALL run the oMLX CLI; +`vllm` SHALL run `vllm serve`, with the model passed as its positional +argument, the served name as `--served-model-name`, and the context window as +`--max-model-len`; `mtplx` SHALL run `mtplx serve`, with the model as +`--model`, the served name as `--model-id`, the context window as +`--context-window`, and a `--download` flag so the engine fetches a model it +does not have itself. There SHALL be no default: a `PROVIDER` that is not a +self-hosted engine SHALL fail, naming the providers that can be served, rather +than launching an engine the Spinloop did not ask for. + +An engine whose executable is normally installed outside the `PATH` SHALL also be +looked for at its conventional install location, so a user who has never put it +on their `PATH` can still serve. + +#### Scenario: Provider selects the engine + +- **WHEN** the Spinloop says `PROVIDER omlx` +- **THEN** the printed command runs the oMLX CLI, not `llama-server` + +#### Scenario: vLLM is servable + +- **WHEN** the Spinloop says `PROVIDER vllm` with a `MODEL` +- **THEN** the printed command runs `vllm serve` with that model as the + positional argument + +#### Scenario: MTPLX is servable + +- **WHEN** the Spinloop says `PROVIDER mtplx` with a `MODEL` naming an MTPLX + optimised repo or a local path +- **THEN** the printed command runs `mtplx serve` with that model as `--model` + and includes `--download` + +#### Scenario: MTPLX served name and context + +- **WHEN** the Spinloop says `PROVIDER mtplx` with `ALIAS qwen` and + `CONTEXT 128k` +- **THEN** the printed command carries `--model-id qwen` and + `--context-window 128000` + +#### Scenario: A provider that is not a local engine + +- **WHEN** the Spinloop names a hosted provider such as `ollama` or `openrouter` +- **THEN** the command fails, listing the providers that can be served locally + +#### Scenario: Engine installed outside the PATH + +- **WHEN** the oMLX CLI is not on the `PATH` but is present in its macOS app + bundle +- **THEN** `serve` uses the bundled executable + +### Requirement: Parallelism + +`CONTEXT` SHALL always mean the context window a single request gets, whatever +engine serves it. A Spinloop's `PARALLEL` SHALL set the number of concurrent +request slots, translated per engine so that `CONTEXT`'s meaning holds: + +- For `llamacpp`, `PARALLEL n` SHALL be rendered as `--parallel n`. Because + llama.cpp treats `--ctx-size` as a total budget divided across its parallel + slots, when the Spinloop **also** states `CONTEXT`, the rendered `--ctx-size` + SHALL be `context_tokens * n` rather than `context_tokens`, so each slot + still gets the context the Spinloop asked for. `CONTEXT` set with no + `PARALLEL` SHALL render `--ctx-size` as `context_tokens`, unscaled, exactly + as before this capability existed. +- For `vllm`, `PARALLEL n` SHALL be rendered as `--max-num-seqs n`. + `--max-model-len` (from `CONTEXT`) SHALL NOT be scaled by `PARALLEL`: vLLM's + concurrency is bounded independently of a single request's context length. +- For `omlx`, `PARALLEL n` SHALL be rendered as `--max-concurrent-requests n`. + oMLX has no context flag to scale either way. +- For `mtplx`, `PARALLEL n` SHALL be rendered as `--max-active-requests n`, + MTPLX's cap on admitted concurrent requests. `--context-window` (from + `CONTEXT`) SHALL NOT be scaled by `PARALLEL`. MTPLX's scheduling mode — how + admitted requests execute, serially or in parallel — SHALL remain a preset + concern: a Spinloop's `PARALLEL` SHALL NOT select it. + +A Spinloop stating no `PARALLEL` SHALL produce a command identical to one from +before this capability existed, for all four engines. A `PARALLEL` value +SHALL be validated as a positive integer at the point it is used (`serve`, a +daemon-pushed config, or `remote deploy`); a value that is not SHALL fail +naming the invalid value rather than being passed to the engine. + +When `CONTEXT` is supplied by a `PRESET` section's own `ctx-size` rather than +by the Spinloop, `PARALLEL` SHALL NOT rescale it — only a Spinloop-stated +`CONTEXT` participates in the `llamacpp` multiply. A `PARALLEL` value SHALL +still override a preset's own `np`/`parallel` (llama.cpp), `max-num-seqs` +(vLLM), `max-concurrent-requests` (oMLX), or `max-active-requests` (MTPLX) +value by the same override-by-canonical-name rule `CONTEXT` already uses +against a preset's `ctx-size`. + +#### Scenario: llama.cpp context is scaled by parallel slots + +- **WHEN** a Spinloop states `PROVIDER llamacpp`, `CONTEXT 128k`, and + `PARALLEL 2` +- **THEN** the printed command includes `--ctx-size 256000` and `--parallel 2` + +#### Scenario: llama.cpp parallel with no context set + +- **WHEN** a Spinloop states `PROVIDER llamacpp` and `PARALLEL 4` with no + `CONTEXT` +- **THEN** the printed command includes `--parallel 4` and no `--ctx-size` + flag is added + +#### Scenario: llama.cpp context with no parallel set is unscaled + +- **WHEN** a Spinloop states `PROVIDER llamacpp` and `CONTEXT 128k` with no + `PARALLEL` +- **THEN** the printed command includes `--ctx-size 128000`, exactly as it + would have before `PARALLEL` existed + +#### Scenario: A preset's own context is not rescaled + +- **WHEN** a Spinloop states `PROVIDER llamacpp`, `PARALLEL 2`, and a `PRESET` + whose section sets `ctx-size` but the Spinloop itself states no `CONTEXT` +- **THEN** the printed command includes `--parallel 2` and the preset's + `ctx-size` value passes through unscaled + +#### Scenario: vLLM concurrency does not touch context + +- **WHEN** a Spinloop states `PROVIDER vllm`, `CONTEXT 128k`, and `PARALLEL 4` +- **THEN** the printed command includes `--max-model-len 128000` and + `--max-num-seqs 4`, with neither value derived from the other + +#### Scenario: oMLX concurrency + +- **WHEN** a Spinloop states `PROVIDER omlx` and `PARALLEL 8` +- **THEN** the printed command includes `--max-concurrent-requests 8` and no + context flag, exactly as with no `PARALLEL` stated + +#### Scenario: MTPLX concurrency does not touch context + +- **WHEN** a Spinloop states `PROVIDER mtplx`, `CONTEXT 128k`, and + `PARALLEL 4` +- **THEN** the printed command includes `--max-active-requests 4` and + `--context-window 128000`, with neither value derived from the other + +#### Scenario: No PARALLEL means no change + +- **WHEN** a Spinloop states no `PARALLEL`, for any of the four engines +- **THEN** the printed command is identical to what it would have been before + this capability existed + +#### Scenario: Invalid parallel count + +- **WHEN** a Spinloop states `PARALLEL 0`, a negative number, or a non-numeric + value, for any of the four engines +- **THEN** the command fails naming the invalid value rather than passing it + to the engine + +#### Scenario: Spinloop PARALLEL overrides a preset's own value + +- **WHEN** a Spinloop states `PROVIDER llamacpp` and `PARALLEL 2`, and its + `PRESET` section separately sets `np = 4` +- **THEN** the printed command includes `--parallel 2`, not `--parallel 4` diff --git a/openspec/changes/archive/2026-09-04-add-mtplx-engine/specs/provider-catalog/spec.md b/openspec/changes/archive/2026-09-04-add-mtplx-engine/specs/provider-catalog/spec.md new file mode 100644 index 00000000..afc2cb46 --- /dev/null +++ b/openspec/changes/archive/2026-09-04-add-mtplx-engine/specs/provider-catalog/spec.md @@ -0,0 +1,31 @@ +## ADDED Requirements + +### Requirement: MTPLX local provider + +The catalogue SHALL include an `mtplx` provider describing a local +[MTPLX](https://mtplx.com) server: an OpenAI-compatible endpoint defaulting to +`http://localhost:8000/v1`, overridable per-provider with `MTPLX_BASE_URL`, and +Pi-capable through the `openai-completions` API. + +Like the llama.cpp and oMLX providers it SHALL be optionally authenticated: a +local server needs no key, while the same provider may name an MTPLX server +started with `--api-key` on another machine. An unset key at a local endpoint +therefore yields no `apiKey` option for opencode and the keyless placeholder +for Pi, while a set key — or any non-local endpoint — yields the environment +reference the harness resolves at run time. + +Its default port coinciding with another provider's is not a conflict: the +catalogue places no uniqueness requirement on base URLs. + +#### Scenario: Local server needs no key + +- **WHEN** a selection names `mtplx` with the default localhost base URL and no + API key set +- **THEN** the opencode provider block carries no `apiKey` option, and the Pi + entry carries the keyless placeholder so its models stay selectable + +#### Scenario: Remote server keeps the key reference + +- **WHEN** the base URL is overridden to a non-local host +- **THEN** the API key is written as an environment reference, never as the + resolved secret diff --git a/openspec/changes/archive/2026-09-04-add-mtplx-engine/tasks.md b/openspec/changes/archive/2026-09-04-add-mtplx-engine/tasks.md new file mode 100644 index 00000000..74407793 --- /dev/null +++ b/openspec/changes/archive/2026-09-04-add-mtplx-engine/tasks.md @@ -0,0 +1,37 @@ +# Tasks + +## 1. Engine: the mtplx entry (cmd/spinloop/serve.go) + +- [x] 1.1 Add the `mtplx` case to `engineFor`: `--model`, `--model-id`, `--context-window`, `--max-active-requests`, `--host`/`--port` (omitted when no BASEURL), `--download` always passed, `--api-key-file` key gating, `needsModel`, loopback default bind, default base URL `http://127.0.0.1:8000` +- [x] 1.2 Serve tests: model, alias, context, parallel, baseurl rendered; baseurl omitted entirely when unset; `--download` present; a set key never on the command line +- [x] 1.3 Confirm the zero preset dialect renders mtplx preset keys verbatim (long-form passthrough, no alias table) + +## 2. Readiness (internal/daemon/readiness.go + the address plumbing) + +- [x] 2.1 Add `mtplx` to `readinessCheckedRunners`, decoupling the engine's address from its metrics dialect so a dialect-less engine can still be probed at `/health` (see design, "Readiness") +- [x] 2.2 Tests: an open mtplx engine reports ready, a gated one (401) counts as ready, an unreachable one does not; the sampler skips a dialect-less target; `scrapeTargetFor` resolves an mtplx address with an empty dialect + +## 3. Catalogue (internal/catalog/providers.yaml) + +- [x] 3.1 Add the `mtplx` provider: `http://localhost:8000/v1` default, `MTPLX_BASE_URL` override, optional `OPENAI_API_KEY`, Pi `openai-completions`, the lucinate marker +- [x] 3.2 Tests: opencode with no key (no `apiKey` option), opencode with a key (environment reference, never the secret), Pi keyless placeholder, lucinate accepted + +## 4. Wake: the runner-aware deploy-config path (cmd/spinloop/remote.go) + +- [x] 4.1 Move the runner gate to the deploy target: the cloud keeps `llamacpp`/`vllm` and its error message; the node path accepts `llamacpp`, `vllm`, `mtplx` +- [x] 4.2 Per-runner preset fallback keys: model `hf` (llamacpp, vllm) / `model` (mtplx); context `ctx-size` (llamacpp, vllm) / `context-window` (mtplx) +- [x] 4.3 Refuse a local model path only where the destination seeds the weights (the cloud); the node path carries the path as the model to load — unblocking llamacpp and vllm node wakes with local weights as a side effect +- [x] 4.4 Add `model-id`, `context-window`, `max-active-requests` to the owned preset keys, and the `mtplx: "max-active-requests"` case to `parallelPresetKey` +- [x] 4.5 Tests: an mtplx wake from a Spinloop with and without a preset (per-runner fallbacks read back), an mtplx wake with a local model path, an llamacpp wake with a local model path now succeeding, the cloud still refusing both mtplx and local paths with today's messages + +## 5. Docs and example + +- [x] 5.1 `docs/commands/serve.md` and `docs/spinloop-file.md`: mtplx as a servable engine, its flag mappings, `--download`, and the scheduling mode staying in presets +- [x] 5.2 `docs/commands/remote.md`: mtplx is not a cloud runner (Apple-Silicon-only, no machine image) +- [x] 5.3 `docs/openapi.yaml`: add `mtplx` to the runner examples (`omlx` is absent too — add it while here) +- [x] 5.4 `examples/mtplx/`: a README and a `Spinloop`, mirroring `examples/omlx/` + +## 6. Verify + +- [x] 6.1 `gofmt -w ./...`, `go vet ./...`, `go test ./... -cover` (total coverage stays at 80% or above) +- [x] 6.2 `openspec change validate add-mtplx-engine` diff --git a/openspec/specs/fleet-routing/spec.md b/openspec/specs/fleet-routing/spec.md index 9a41f0ec..f05766ac 100644 --- a/openspec/specs/fleet-routing/spec.md +++ b/openspec/specs/fleet-routing/spec.md @@ -203,6 +203,13 @@ only that a process exists, but until its engine endpoint answers. A node whose stored config already matches the wanted model SHALL be preferred, since it has the weights. +The pushed config is the node-side counterpart of what `spinloop serve` would run +for that Spinloop, translated per engine. A node may be woken for an engine that +binds its model at launch — `llamacpp`, `vllm`, and `mtplx` — and a `MODEL` that +names a file on the node's own disk is a valid wake for it: the node has the +file, and only a destination that fetches its weights itself refuses a local +path. + A node that refuses the config — a runner or model it cannot serve — SHALL NOT fail the launch while other candidates remain: the next candidate SHALL be tried, and the refusals SHALL be reported when none succeeds. @@ -228,6 +235,19 @@ command that would start one. - **THEN** that node is given the Spinloop's model as its deploy config, started, and the agent launches against it once its engine answers +#### Scenario: A node is woken for a Mac-only engine + +- **WHEN** a fleet-routed launch finds no node serving the wanted model, and an + idle node's daemon can run MTPLX +- **THEN** that node is woken with a config that runs the wanted model under + `mtplx serve`, and the agent launches against it once its engine answers + +#### Scenario: A local model path wakes the node that has it + +- **WHEN** the Spinloop's `MODEL` names a file on the woken node's disk +- **THEN** the wake carries that path as the model to load, rather than + refusing it as a local file + #### Scenario: A started engine that is not yet loaded is waited for - **WHEN** a woken node reports `running` while its engine is still loading diff --git a/openspec/specs/local-serving/spec.md b/openspec/specs/local-serving/spec.md index 5efe37d0..0a327ddf 100644 --- a/openspec/specs/local-serving/spec.md +++ b/openspec/specs/local-serving/spec.md @@ -34,7 +34,10 @@ the local counterpart of the runner `spinloop remote deploy` selects from the sa instruction. `llamacpp` SHALL run `llama-server`; `omlx` SHALL run the oMLX CLI; `vllm` SHALL run `vllm serve`, with the model passed as its positional argument, the served name as `--served-model-name`, and the context window as -`--max-model-len`. There SHALL be no default: a `PROVIDER` that is not a +`--max-model-len`; `mtplx` SHALL run `mtplx serve`, with the model as +`--model`, the served name as `--model-id`, the context window as +`--context-window`, and a `--download` flag so the engine fetches a model it +does not have itself. There SHALL be no default: a `PROVIDER` that is not a self-hosted engine SHALL fail, naming the providers that can be served, rather than launching an engine the Spinloop did not ask for. @@ -53,6 +56,20 @@ on their `PATH` can still serve. - **THEN** the printed command runs `vllm serve` with that model as the positional argument +#### Scenario: MTPLX is servable + +- **WHEN** the Spinloop says `PROVIDER mtplx` with a `MODEL` naming an MTPLX + optimised repo or a local path +- **THEN** the printed command runs `mtplx serve` with that model as `--model` + and includes `--download` + +#### Scenario: MTPLX served name and context + +- **WHEN** the Spinloop says `PROVIDER mtplx` with `ALIAS qwen` and + `CONTEXT 128k` +- **THEN** the printed command carries `--model-id qwen` and + `--context-window 128000` + #### Scenario: A provider that is not a local engine - **WHEN** the Spinloop names a hosted provider such as `ollama` or `openrouter` @@ -216,9 +233,14 @@ request slots, translated per engine so that `CONTEXT`'s meaning holds: concurrency is bounded independently of a single request's context length. - For `omlx`, `PARALLEL n` SHALL be rendered as `--max-concurrent-requests n`. oMLX has no context flag to scale either way. +- For `mtplx`, `PARALLEL n` SHALL be rendered as `--max-active-requests n`, + MTPLX's cap on admitted concurrent requests. `--context-window` (from + `CONTEXT`) SHALL NOT be scaled by `PARALLEL`. MTPLX's scheduling mode — how + admitted requests execute, serially or in parallel — SHALL remain a preset + concern: a Spinloop's `PARALLEL` SHALL NOT select it. A Spinloop stating no `PARALLEL` SHALL produce a command identical to one from -before this capability existed, for all three engines. A `PARALLEL` value +before this capability existed, for all four engines. A `PARALLEL` value SHALL be validated as a positive integer at the point it is used (`serve`, a daemon-pushed config, or `remote deploy`); a value that is not SHALL fail naming the invalid value rather than being passed to the engine. @@ -227,9 +249,9 @@ When `CONTEXT` is supplied by a `PRESET` section's own `ctx-size` rather than by the Spinloop, `PARALLEL` SHALL NOT rescale it — only a Spinloop-stated `CONTEXT` participates in the `llamacpp` multiply. A `PARALLEL` value SHALL still override a preset's own `np`/`parallel` (llama.cpp), `max-num-seqs` -(vLLM), or `max-concurrent-requests` (oMLX) value by the same -override-by-canonical-name rule `CONTEXT` already uses against a preset's -`ctx-size`. +(vLLM), `max-concurrent-requests` (oMLX), or `max-active-requests` (MTPLX) +value by the same override-by-canonical-name rule `CONTEXT` already uses +against a preset's `ctx-size`. #### Scenario: llama.cpp context is scaled by parallel slots @@ -270,16 +292,23 @@ override-by-canonical-name rule `CONTEXT` already uses against a preset's - **THEN** the printed command includes `--max-concurrent-requests 8` and no context flag, exactly as with no `PARALLEL` stated +#### Scenario: MTPLX concurrency does not touch context + +- **WHEN** a Spinloop states `PROVIDER mtplx`, `CONTEXT 128k`, and + `PARALLEL 4` +- **THEN** the printed command includes `--max-active-requests 4` and + `--context-window 128000`, with neither value derived from the other + #### Scenario: No PARALLEL means no change -- **WHEN** a Spinloop states no `PARALLEL`, for any of the three engines +- **WHEN** a Spinloop states no `PARALLEL`, for any of the four engines - **THEN** the printed command is identical to what it would have been before this capability existed #### Scenario: Invalid parallel count - **WHEN** a Spinloop states `PARALLEL 0`, a negative number, or a non-numeric - value, for any of the three engines + value, for any of the four engines - **THEN** the command fails naming the invalid value rather than passing it to the engine diff --git a/openspec/specs/provider-catalog/spec.md b/openspec/specs/provider-catalog/spec.md index e4f5e0dc..1cebd5e5 100644 --- a/openspec/specs/provider-catalog/spec.md +++ b/openspec/specs/provider-catalog/spec.md @@ -195,3 +195,33 @@ catalogue places no uniqueness requirement on base URLs. - **THEN** the API key is written as an environment reference, never as the resolved secret +### Requirement: MTPLX local provider + +The catalogue SHALL include an `mtplx` provider describing a local +[MTPLX](https://mtplx.com) server: an OpenAI-compatible endpoint defaulting to +`http://localhost:8000/v1`, overridable per-provider with `MTPLX_BASE_URL`, and +Pi-capable through the `openai-completions` API. + +Like the llama.cpp and oMLX providers it SHALL be optionally authenticated: a +local server needs no key, while the same provider may name an MTPLX server +started with `--api-key` on another machine. An unset key at a local endpoint +therefore yields no `apiKey` option for opencode and the keyless placeholder +for Pi, while a set key — or any non-local endpoint — yields the environment +reference the harness resolves at run time. + +Its default port coinciding with another provider's is not a conflict: the +catalogue places no uniqueness requirement on base URLs. + +#### Scenario: Local server needs no key + +- **WHEN** a selection names `mtplx` with the default localhost base URL and no + API key set +- **THEN** the opencode provider block carries no `apiKey` option, and the Pi + entry carries the keyless placeholder so its models stay selectable + +#### Scenario: Remote server keeps the key reference + +- **WHEN** the base URL is overridden to a non-local host +- **THEN** the API key is written as an environment reference, never as the + resolved secret +