Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 12 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,6 +57,7 @@ their models.
| llama.cpp | `llamacpp` | Local (or remote) llama-server |
| oMLX | `omlx` | Local oMLX server on Apple Silicon |
| vLLM | `vllm` | Local or self-hosted vLLM server |
| MTPLX | `mtplx` | Local MTPLX server on Apple Silicon |
| OpenAI-compatible | `openai-compatible` | Any endpoint that speaks the OpenAI API — set the base URL and key |

Adding one that isn't here is a data change, not code — see
Expand All @@ -80,7 +81,8 @@ model refs and quirks, and none of it is written down where you need it.
it configures the agent for you:

- **One command, any model.** Pick from a built-in catalogue — OpenRouter,
Bedrock, Ollama, llama.cpp, vLLM, oMLX, or any OpenAI-compatible endpoint. No
Bedrock, Ollama, llama.cpp, vLLM, oMLX, MTPLX, or any OpenAI-compatible
endpoint. No
URLs to look up, and `spinloop list --models` fetches the model ids straight from
the provider.
- **Your config survives.** Settings are merged *into* what you already have.
Expand Down Expand Up @@ -353,8 +355,9 @@ the bash and zsh completions for you.

Running a model locally? `spinloop serve` reads a `Spinloop` and launches the
inference server its `PROVIDER` names — `llamacpp` runs `llama-server`, `omlx`
runs [oMLX](https://omlx.ai) on Apple Silicon — so the same file that points
opencode at a model can start it too. The simple case needs no preset:
runs [oMLX](https://omlx.ai) and `mtplx` runs [MTPLX](https://mtplx.com), the
two Apple-Silicon engines — so the same file that points opencode at a model can
start it too. The simple case needs no preset:

```dockerfile
# Spinloop
Expand All @@ -376,8 +379,8 @@ the chosen section into the command instead — with anything the `Spinloop` sta
(like `CONTEXT`) overriding the preset. It's the missing piece presets don't
cover: launching a *single* model. `CONTEXT` always means the context per
request; add `PARALLEL` to run more than one slot and `spinloop` works out each
engine's own accounting (llama.cpp's `--ctx-size` gets scaled, vLLM's and
oMLX's don't) — see
engine's own accounting (llama.cpp's `--ctx-size` gets scaled; vLLM's and
MTPLX's don't, and oMLX has none) — see
[Parallelism](docs/commands/serve.md#parallelism) for the full mapping.
Details in [`docs/commands/serve.md`](docs/commands/serve.md).

Expand Down Expand Up @@ -582,8 +585,8 @@ machine; it is never sent to the deployed instance.
Each provider declares which environment variable holds its key (`spinloop
list` shows them). Values are looked up in your shell environment first, then a
`.env` beside the `Spinloop`, so an exported variable always wins and the `.env`
only fills a gap. Local providers like Ollama, llama.cpp and oMLX need no
key;
only fills a gap. Local providers like Ollama, llama.cpp, oMLX and MTPLX need
no key;
Bedrock authenticates through your AWS credentials.

`spinloop harness` carries that same local environment to the agent it launches:
Expand All @@ -603,7 +606,7 @@ SPINLOOP_BASE_URL=https://gateway/v1 spinloop add -p openai-compatible -m my-mod

The flag wins over the env var, and either wins over the catalogue's defaults
and the per-provider variables (`OLLAMA_BASE_URL`, `LLAMACPP_BASE_URL`,
`OMLX_BASE_URL`, `VLLM_BASE_URL`, `OPENAI_BASE_URL`).
`OMLX_BASE_URL`, `VLLM_BASE_URL`, `MTPLX_BASE_URL`, `OPENAI_BASE_URL`).

## Guides

Expand All @@ -616,6 +619,7 @@ with a ready-to-apply `Spinloop`:
- [Gemma-4-12B-IT on llama.cpp](examples/llamacpp/gemma4/README.md)
- [Qwen3.6-35B-A3B on oMLX (Apple Silicon)](examples/omlx/qwen3.6/README.md)
- [Gemma-4-E2B on oMLX (Apple Silicon)](examples/omlx/gemma-4-e2b/README.md)
- [Qwen3.8-27B on MTPLX (Apple Silicon)](examples/mtplx/qwen3.8-27b/README.md)
- [Fetching a Spinloop from a URL](examples/remote-spinloop/README.md)

## Adding providers and models
Expand Down
85 changes: 73 additions & 12 deletions cmd/spinloop/remote.go
Original file line number Diff line number Diff line change
Expand Up @@ -1050,6 +1050,21 @@ func runnerFor(provider string) (string, error) {
}
}

// nodeRunnerFor is the runner resolver for the node path — waking a fleet node
// that already exists. It accepts every engine `serve` can run and a daemon can
// supervise: llamacpp, vllm, and mtplx. MTPLX is Apple-Silicon-only and has no
// machine image, so it never becomes a cloud runner (see runnerFor).
func nodeRunnerFor(provider string) (string, error) {
switch provider {
case "llamacpp", "vllm", "mtplx":
return provider, nil
default:
return "", fmt.Errorf(
"PROVIDER %q cannot be woken: a fleet node runs a self-hosted engine, so use llamacpp, vllm or mtplx",
provider)
}
}

// splitModelQuant splits a model reference into the Hugging Face repo and an
// optional quant tag, as used by llama.cpp's -hf (org/model:QUANT). Repo ids
// cannot contain a colon, so the first one separates them.
Expand All @@ -1072,6 +1087,12 @@ var cloudOwnedFlags = map[string]bool{
"hf-repo": true, "hf-file": true, "hf-token": true,
"api-key": true, "api-key-file": true,
"ctx-size": true, "alias": true, "metrics": true,
// MTPLX's own spellings of the settings the destination computes from the
// deploy config: the served name, the context window, and the slot count.
// Keyed by canonical name like the rest, so a preset's raw values do not
// double-define the computed flags. They collide with no llama.cpp or vLLM
// key, so the cloud path is untouched.
"model-id": true, "context-window": true, "max-active-requests": true,
// Companion weights: the cloud syncs these from S3 and names them at its
// own paths, so the preset's local paths must not travel. Only the
// location is cloud-owned — how the engine is asked to use a drafter
Expand Down Expand Up @@ -1111,6 +1132,35 @@ func parallelPresetKey(runner string) string {
return "max-num-seqs"
case "omlx":
return "max-concurrent-requests"
case "mtplx":
return "max-active-requests"
default:
return ""
}
}

// modelPresetKey names the preset key holding the model, in that runner's own
// vocabulary — the same split the *ServeParams functions make when they read
// it. A runner with no preset model key yields "", which reads as "not set".
func modelPresetKey(runner string) string {
switch runner {
case "mtplx":
return "model"
case "llamacpp", "vllm":
return "hf"
default:
return ""
}
}

// contextPresetKey names the preset key holding the context window, in that
// runner's own vocabulary.
func contextPresetKey(runner string) string {
switch runner {
case "mtplx":
return "context-window"
case "llamacpp", "vllm":
return "ctx-size"
default:
return ""
}
Expand Down Expand Up @@ -1157,6 +1207,7 @@ func dropOwned(owned func(string) bool, params []preset.Param) []preset.Param {
// caller where that is not true.
func deployConfigFor(sel spinloop.Selection, spinloopPath string) (remote.DeployConfig, error) {
return deployConfig(sel, spinloopPath, deployTarget{
runner: runnerFor,
requireContext: true,
seedsWeights: true,
owns: isCloudOwned,
Expand All @@ -1170,13 +1221,18 @@ func deployConfigFor(sel spinloop.Selection, spinloopPath string) (remote.Deploy
// that `spinloop serve` runs happily needs no CONTEXT added merely to be routed;
// and the preset's bind survives, so an engine told to listen on 0.0.0.0 does.
func deployConfigForNode(sel spinloop.Selection, spinloopPath string) (remote.DeployConfig, error) {
return deployConfig(sel, spinloopPath, deployTarget{owns: isNodeOwned})
return deployConfig(sel, spinloopPath, deployTarget{
runner: nodeRunnerFor,
owns: isNodeOwned,
})
}

// deployTarget is what the derivation cannot decide for itself: whether a
// context size is required, which preset flags the destination assigns, and
// whether it fetches the weights itself (and so needs companions named).
// deployTarget is what the derivation cannot decide for itself: which runners
// it accepts, whether a context size is required, which preset flags the
// destination assigns, and whether it fetches the weights itself (and so needs
// companions named).
type deployTarget struct {
runner func(provider string) (string, error)
requireContext bool
seedsWeights bool
owns func(key string) bool
Expand All @@ -1185,7 +1241,7 @@ type deployTarget struct {
func deployConfig(sel spinloop.Selection, spinloopPath string, target deployTarget) (remote.DeployConfig, error) {
var dc remote.DeployConfig

runner, err := runnerFor(sel.Provider)
runner, err := target.runner(sel.Provider)
if err != nil {
return dc, err
}
Expand Down Expand Up @@ -1219,14 +1275,19 @@ func deployConfig(sel spinloop.Selection, spinloopPath string, target deployTarg

model := sel.Model
if model == "" {
model = presetValue("hf", global, params)
model = presetValue(modelPresetKey(dc.Runner), global, params)
}
if model == "" {
return dc, fmt.Errorf(
"nothing to deploy: set MODEL (an HF repo like org/model:QUANT) in %s, or hf in its preset",
spinloopPath)
}
if isModelPath(model) {
"nothing to deploy: set MODEL in %s, or %s in its preset",
spinloopPath, modelPresetKey(dc.Runner))
}
// A local model path is refused only where the destination fetches the
// weights itself — the cloud, which cannot ship a file. A node has the file
// the Spinloop points at, so the node path carries the path as the model to
// load; today's unconditional check also blocked llamacpp and vllm node
// wakes with local weights, which moving it unblocks too.
if target.seedsWeights && isModelPath(model) {
return dc, fmt.Errorf(
"cannot deploy the local model file %q: the cloud downloads weights from Hugging Face, so name a repo (org/model:QUANT)",
model)
Expand All @@ -1235,10 +1296,10 @@ func deployConfig(sel spinloop.Selection, spinloopPath string, target deployTarg

context := sel.Context
if context == "" {
context = presetValue("ctx-size", global, params)
context = presetValue(contextPresetKey(dc.Runner), global, params)
}
if context == "" && target.requireContext {
return dc, fmt.Errorf("no context size: set CONTEXT in %s, or ctx-size in its preset", spinloopPath)
return dc, fmt.Errorf("no context size: set CONTEXT in %s, or %s in its preset", spinloopPath, contextPresetKey(dc.Runner))
}
if context != "" {
n, err := contextsize.Parse(context)
Expand Down
Loading