Huggingface Hub Cache Support

[MCJEModder2026]

Problem

built in model downloader is very slow (10-20 mbps vs hf cli's 200 mbps)
in order to use hf cache models i have to go into ~/.cache/huggingface/hub and get the path which is annoying
it does not let me use already downloaded models for forging into mtplx

Proposal

use huggingface hub cache folder for model downloads

[wparuch]

Environment for everything below

MacBook Pro, M2 Max, 32 GB, macOS 26.6.2
mtplx 2.11.1 (pip, isolated venv) · mlx 0.32.2 · mlx-lm 0.31.3
models involved: Youssofal/Qwen3.8-27B-MTPLX-Bare-Speed-FP16 (16.3 GB)
                 Youssofal/Qwen3.8-27B-MTPLX-Optimized-Speed-FP16 (20.7 GB)
                 mlx-community/Qwen3-Embedding-4B-4bit-DWQ (2.1 GB)

---

1. The private cache duplicates weights, and a 27B pair is enough to fill a disk

~/.mtplx/models is a second, parallel model store. On this machine the two 27B builds
came to 37 GB, while ~/.cache/huggingface/hub already held the plain
mlx-community/Qwen3.8-27B-4bit and several other checkpoints of the same family.

This is not a per-user tidiness complaint. Anyone evaluating MTPLX against another MLX
runtime necessarily has the comparison checkpoints in the HF cache already, so the disk
cost of trying MTPLX is double what the model actually weighs.

2. The failure mode is a runtime 500, not a startup error

This is the part worth fixing first, independently of where the cache lives.

Retrieval models are resolved lazily, so a model that is not in ~/.mtplx/models
starts the server successfully and is advertised as available:

mtplx serve --model Youssofal/Qwen3.8-27B-MTPLX-Bare-Speed-FP16 \
  --embedding-model mlx-community/Qwen3-Embedding-4B-4bit-DWQ --port 8080
# → "MTPLX is ready."

curl -s "http://127.0.0.1:8080/v1/models?capability=embedding"
# → {"data":[{"id":"Qwen3-Embedding-4B-4bit-DWQ","capability":"embedding", ...}]}

The model is listed. Everything looks configured. Then the first real request:

curl -s http://127.0.0.1:8080/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3-Embedding-4B-4bit-DWQ","input":["hello"]}'
# → HTTP 500

unhandled server error path=/v1/embeddings:
  Model mlx-community/Qwen3-Embedding-4B-4bit-DWQ is not cached.
  Run: mtplx pull mlx-community/Qwen3-Embedding-4B-4bit-DWQ
Traceback (most recent call last):
  File "mtplx/server/openai.py", line 29815, in embeddings
  File "mtplx/retrieval.py", line 795, in embed
  File "mtplx/retrieval.py", line 676, in _acquire
  File "mtplx/retrieval.py", line 660, in _backend_key
    raise FileNotFoundError(...)

In production this surfaces as a 500 on a user's first question, not as a failure to
boot. Two cheap fixes, both worth having even if the cache question is deferred:

• validate --embedding-model / --reranker-model at startup and refuse to start
  (the same way an unusable chat model is refused), or
• do not list a retrieval model in /v1/models until it is known to be loadable,
  and return a 4xx with the mtplx pull hint instead of a 500 traceback.

The error text itself is good — it names the exact command. It just arrives at the
worst possible moment.

3. Forge cannot consume an already-downloaded checkpoint

Same root cause: the resolver only looks in the private cache, so a source checkpoint
sitting in the HF cache has to be re-downloaded before it can be forged. For a 27B
source that is a second copy of 15–30 GB on disk for no benefit.

---

Workarounds (all three verified on 2.11.1)

A. Symlink the HF snapshot into the MTPLX cache — recommended, zero duplication

SNAP=$(ls -d ~/.cache/huggingface/hub/models--mlx-community--Qwen3-Embedding-4B-4bit-DWQ/snapshots/*/ | head -1)
ln -sfn "${SNAP%/}" ~/.mtplx/models/mlx-community--Qwen3-Embedding-4B-4bit-DWQ

The models--<org>--<name> → <org>--<name> rename is the whole trick. After it,
MTPLX resolves the plain repo id everywhere:

mtplx models
# - mlx-community/Qwen3-Embedding-4B-4bit-DWQ  2.3 GB  contract=false

mtplx inspect mlx-community/Qwen3-Embedding-4B-4bit-DWQ
# tier: AR-only   can_run: true

mtplx serve --model <chat-model> --embedding-model mlx-community/Qwen3-Embedding-4B-4bit-DWQ
curl -s http://127.0.0.1:8080/v1/embeddings -H 'Content-Type: application/json' \
  -d '{"model":"Qwen3-Embedding-4B-4bit-DWQ","input":["hello"]}'
# → 200, dim 2560

du -sh on the link target confirms 2.1 GB is still stored exactly once.

That this works is the strongest argument that the fix is small: nothing in the
loader needs the private directory — it only needs the path resolver to look in the
HF cache too.

B. Pass the snapshot path with an alias — no symlink, but the path is ugly

mtplx serve --model <chat-model> \
  --embedding-model ~/.cache/huggingface/hub/models--mlx-community--Qwen3-Embedding-4B-4bit-DWQ/snapshots/<sha>=qwen3-embed-4b

The REF=served-id form is what keeps this usable — without the alias the served model
id becomes the snapshot sha, which then has to be hardcoded in every client. Worth
documenting in the README next to the retrieval-endpoint example, whatever happens with
the cache proposal.

C. mtplx pull — works, costs a second copy

mtplx pull mlx-community/Qwen3-Embedding-4B-4bit-DWQ

---

Proposal, concretely

Resolve a repo id through huggingface_hub before falling back to the private cache:

from huggingface_hub import snapshot_download
path = snapshot_download(repo_id, local_files_only=True)   # raises if not cached

snapshot_download(..., local_files_only=True) already honours HF_HOME,
HF_HUB_CACHE and HF_HUB_OFFLINE, and returns the same snapshot directory the
symlink in workaround A points at — so the change is in _backend_key /
the model-path resolver, not in the loaders.

Suggested order of resolution, so nothing currently working breaks:

1. explicit local path (already supported)
2. ~/.mtplx/models/<org>--<name> (current behaviour, keeps every existing install)
3. the HF cache via snapshot_download(local_files_only=True) — new
4. download (into whichever store, ideally the HF cache)

The same resolver would fix Forge's inability to consume an existing checkpoint, since
that is the same lookup.

If the private cache exists to guarantee an artifact was admitted through MTPLX's own
validation, note that the validation already runs independently of location — workaround A
shows mtplx inspect classifying a symlinked HF snapshot correctly (tier: AR-only,
can_run: true), and the runtime-contract check on a forged pack would run the same way.

---

One claim I could not reproduce: downloader throughput

The original report puts the built-in downloader at 10–20 Mbit/s against ~200 Mbit/s for
the hf CLI. On this machine the measurement came out the other way round, so I would
not want it carried into the issue unqualified.

Method: same repo (Youssofal/Qwen3.5-9B-MTPLX-Optimized-Speed, 8.7 GB), cold each time,
alternating, 60 s per run, bytes on disk measured afterwards (for hf counting both the
snapshot directory and the growth of ~/.cache/huggingface/xet):

run	bytes in 60 s	throughput
mtplx pull #1	1130 MB	151 Mbit/s
hf download #1	21 MB	3 Mbit/s
mtplx pull #2	756 MB	101 Mbit/s
hf download #2	21 MB	3 Mbit/s
mtplx pull #3	959 MB	128 Mbit/s

Both clients ran unauthenticated, and hf warned about it every time
(You are sending unauthenticated requests to the HF Hub... set a HF_TOKEN to enable higher rate limits and faster downloads). The ~/.cache/huggingface/xet directory exists
on this machine, so hf was going through Xet, and Xet's CAS negotiation is exactly what
the token affects.

So the likeliest reading is that the original 200 Mbit/s figure is hf + HF_TOKEN +
Xet-backed storage, and the gap is not the raw transfer but whether the client can use
Xet chunked/parallel transfer at all. If so, the actionable form of that bullet is
"support Xet / honour HF_TOKEN in the built-in downloader" rather than "the downloader
is slow" — and adopting the HF cache (the main proposal) gets that for free, because it
would mean using huggingface_hub for transfers as well.

Someone with a token should re-run the table above before this part is treated as fact.

[youssofal]

Thank you both. wparuch's audit turned this into two concrete fixes, both on the branch for the next release.

First, the 500. A retrieval model that is not on disk no longer boots. --embedding-model and --reranker-model are resolved at startup the same way the chat model is, and an unresolvable reference refuses to start with the exact mtplx pull <repo> hint the old traceback carried. /v1/models?capability=embedding lists only the retrieval models that resolved. If something still slips through (a directory removed after boot), /v1/embeddings and /v1/rerank answer a structured 404 with the same hint instead of an unhandled 500.

Second, the cache. resolve_model_path now checks the shared Hugging Face cache between the private store and a download, in exactly the order you proposed: explicit path, then ~/.mtplx/models/<org>--<name>, then huggingface_hub.snapshot_download(repo_id, local_files_only=True), then download. A private-cache hit still wins, so existing installs are unchanged. Forge goes through the same resolver, so a source checkpoint already in the HF cache is no longer downloaded a second time, and the provenance stamped on the built artifact names the copy it was built from. Your symlink workaround still works and is no longer needed. Tests cover the order, the precedence and the fall-through.

On download speed: your alternating measurement (151 versus 3 Mbit/s unauthenticated) is the right way to measure it, and I will not repeat the original figure without that qualification. The useful follow-up is Xet and HF_TOKEN support in the built-in downloader, which follows from using huggingface_hub for transfers. Noted as the next step.