Metal GPU timeout when loading large MLX models from NFS / external volumes; same snapshots work from internal SSD

[JasonRuonanWang]

Summary

I am seeing a reproducible Metal GPU timeout when loading some large MLX-format models from non-internal storage, specifically NFS and a USB external volume.

The exact same model snapshot loads successfully when copied to the internal Mac SSD / APFS volume.

This was first reported downstream in oMLX:

jundot/omlx#2098

I am opening this here because the terminal failure is a fatal MLX/Metal error:

libc++abi: terminating due to uncaught exception of type std::runtime_error:
[METAL] Command buffer execution failed:
Caused GPU Timeout Error (00000002:kIOGPUCommandBufferCallbackErrorTimeout)

Environment

• Machine: Mac Studio, M3 Ultra, 512GB unified memory
• macOS: 26.x
• Downstream server: oMLX
• oMLX versions tested: 0.4.4 and 0.4.5-dev
• Model storage tested:
  • NFS-mounted NAS: fails
  • USB external volume under /Volumes: fails
  • Internal Mac SSD / APFS: works
• NAS connection:
  • 10GbE
  • Smaller models can load from NFS at 600MB/s+ without issue

Models tested

Failing models:

• mlx-community/DeepSeek-V4-Flash-nvfp4
  • discovered size in oMLX: ~148GB
• mlx-community/GLM-5.2-DQ4plus-q8
  • discovered size in oMLX: ~454GB
• Other GLM-5.2 family variants appear affected on the same path

Working control model:

• Qwen/Qwen3-235B-A22B-MLX-8bit
  • discovered size in oMLX: ~237GB

The Qwen model is larger than DeepSeek-V4-Flash-nvfp4 but works from the same environment, so this does not look like a simple total-size, memory-capacity, or storage-bandwidth issue.

Reproduction

Using this model snapshot:

/opt/models/huggingface/hub/models--mlx-community--DeepSeek-V4-Flash-nvfp4/snapshots/55ba3c674ae921984c6842dab393f7f435fb3215

When the snapshot lives on NFS or a USB external volume, loading the model causes a fatal Metal GPU timeout after a few seconds.

When the same snapshot is copied to the internal SSD, resolving HuggingFace cache symlinks:

mkdir -p /Volumes/Cache/DeepSeek
rsync -aL /opt/models/huggingface/hub/models--mlx-community--DeepSeek-V4-Flash-nvfp4/snapshots/55ba3c674ae921984c6842dab393f7f435fb3215/ /Volumes/Cache/DeepSeek/

and then loaded from the internal SSD / APFS path, it works.

Actual behavior

When loading from NFS or USB external storage, the process aborts with:

libc++abi: terminating due to uncaught exception of type std::runtime_error:
[METAL] Command buffer execution failed:
Caused GPU Timeout Error (00000002:kIOGPUCommandBufferCallbackErrorTimeout)

For DeepSeek-V4-Flash-nvfp4 through oMLX, the log shows:

Loading model: mlx-community--DeepSeek-V4-Flash-nvfp4
PoolingCache / BatchPoolingCache injected into mlx_lm.models.cache
mlx_lm.utils.load_model replaced (deepseek_v4 fp8 + F8_E8M0 fallback)
DeepSeek V4 patch applied
DeepSeek-V4 MTP model patch applied
MLX executor thread initialized: generation_stream = ThreadLocalStream(Device(gpu, 0), 4)
libc++abi: terminating due to uncaught exception of type std::runtime_error:
[METAL] Command buffer execution failed:
Caused GPU Timeout Error

For GLM-5.2-DQ4plus-q8 through oMLX, the log shows:

Loading model: mlx-community--GLM-5.2-DQ4plus-q8
GLM MoE DSA native kernels available
GLM MoE DSA optimized mlx-lm patch applied
MLX executor thread initialized: generation_stream = ThreadLocalStream(Device(gpu, 0), 4)
libc++abi: terminating due to uncaught exception of type std::runtime_error:
[METAL] Command buffer execution failed:
Caused GPU Timeout Error

Expected behavior

The model should either:

1. load successfully from NFS / USB external storage, or
2. fail gracefully with a recoverable Python exception instead of aborting the process.

Ideally there should be a safe way to force eager CPU-side page-in / prefault / staging before a long Metal command buffer is submitted.

Hypothesis

This looks like a cold-page / first-touch issue rather than a bandwidth or total-size issue.

The failing cases seem to involve model-family-specific optimized paths / quantized kernels. When the weights are file-backed and live on NFS or external storage, the first GPU-side materialization / dequant / layout work may depend on many cold pages.

If that happens inside one long Metal command buffer, slower or jittery page-in from NFS / USB storage may push the command over the Metal watchdog timeout.

The same path works from internal SSD likely because page-in latency is much lower.

The Qwen3-235B 8bit control model working despite being larger suggests the problem is not simply model size, but the interaction between:

• file-backed MLX arrays / mmap-like loading
• cold pages on slower or remote storage
• initial dequant / materialization / layout kernels
• Metal command buffer timeout behavior

Questions

Could MLX provide guidance on the following?

1. Is loading large file-backed MLX/safetensors weights from NFS or external volumes expected to be supported?
2. Is there a recommended way to prefault or eagerly stage all model weights before first GPU execution?
3. Can MLX split initial materialization / dequant / layout work into smaller command buffers to avoid Metal watchdog timeouts?
4. Is there a way to make this failure recoverable as a Python exception instead of aborting the process?
5. Are there known limitations around Metal command buffers depending on cold file-backed pages?

Related downstream issue

Downstream oMLX issue with full logs and storage-location A/B:

jundot/omlx#2098

[JasonRuonanWang]

Additional storage A/B result:

I tested another external SSD with roughly ~1GB/s sequential read speed.

Result:

• The failure still reproduces.
• The failure mode is the same Metal GPU timeout:
  [METAL] Command buffer execution failed: Caused GPU Timeout Error
• Compared with the NFS case, the time-to-failure is much longer.
  With the faster external SSD, oMLX/MLX keeps reading model weights for roughly ~30 seconds before aborting.
• Formatting the external SSD as APFS vs exFAT does not materially change the behavior. Both formats fail in the same way.

This suggests the issue is probably not specific to NFS protocol semantics or APFS/exFAT filesystem behavior.

The storage medium speed/latency appears to affect how far the loader can progress before the Metal timeout occurs. That strengthens the hypothesis that the failure is related to cold weight pages / first-touch materialization / dequant/layout work being coupled to a long Metal command buffer.

Current observed matrix:

• Internal Mac SSD / APFS: works
• NFS NAS over 10GbE: fails quickly
• External SSD around ~1GB/s: fails later, after roughly ~30s of weight reads
• APFS external SSD: fails
• exFAT external SSD: fails
• Qwen3-235B-A22B-MLX-8bit control model: works, despite being larger than DeepSeek-V4-Flash-nvfp4

So this does not look like a simple total model size issue, a simple bandwidth issue, or a filesystem-format issue. It looks more like an interaction between storage-backed cold pages and the initial MLX/Metal execution path for certain model families / quantization kernels.