CUDA_ERROR_INVALID_HANDLE when evicting memory

[githubnemo]

Hardware:

• 64GB RAM
• 2x 2080 Ti @ 11GB VRAM

Nixie version: compiled from main

The bug:

nixie daemon --shm 20g -l 'g:0.98'
nixie run -d 0,1 ./bin/llama-server -m ~/.cache/llama.cpp/Qwen3.5-35B-A3B-Q4_K_M.gguf -c $((100*1024)) --port 8002 --host 0.0.0.0

works fine but when now also running

nixie run -d 0,1 ./bin/llama-server -m ~/.cache/llama.cpp/ggml-org_Qwen3-8B-GGUF_Qwen3-8B-Q4_K_M.gguf -c $((32*1024)) --port 8003 --host 0.0.0.0

(I don't think the specific model is important; just that it also needs VRAM)

the daemon will report:

10:44:35.849  INFO nixie::runtime::daemon_server: Client[pid=10956, comm="llama-server"] connected             
10:44:35.849 DEBUG nixie::runtime::shm: open_shm(/nixie_ipc-10956-9ca0287a.shm)   
10:44:35.849  INFO nixie::runtime::daemon_server: Client[pid=10956] VRAM limit: ["1 (10597 MB)", "0 (10604 MB)"]                                                              
10:44:35.849  INFO nixie::runtime::proc_ctl: Listen events from process [pid=10956]                                                                                           
10:44:59.021 DEBUG nixie::runtime::schedule::scheduler: Process 10956 requests scheduling                                                                                     
10:44:59.021 DEBUG nixie::runtime::migration::migration_plan: Device GlobalDeviceId(1) requires 0.00 B of data to migrate in
10:44:59.021 DEBUG nixie::runtime::migration::migration_plan: Device GlobalDeviceId(0) requires 0.00 B of data to migrate in                                                  
10:44:59.021 DEBUG nixie::runtime::migration::migration_plan: Created migration task: {"shm -> gpu":{"10956":"0.00 B(0)"}} for 142 us
10:44:59.021 DEBUG nixie::runtime::migration::execution: No migration needed     
10:44:59.022 DEBUG nixie::runtime::schedule::scheduler: Process 10956 requests memory: 1.25 GiB
10:44:59.022 DEBUG nixie::runtime::migration::migration_plan: Device GlobalDeviceId(0) requires 1.25 GiB of data to migrate in                                                
10:44:59.022 DEBUG nixie::runtime::migration::migration_plan: Created migration task: {"gpu -> shm":{"10031":"1.25 GiB(10)"},"shm -> gpu":{"10956":"0.00 B(0)"}} for 113 us
10:44:59.042 DEBUG nixie::runtime::migration::execution: Data migration completed in 0.020s, largest transfer size = 1.25 GiB, speed = 67.965 GB/s                            
10:44:59.177 DEBUG nixie::runtime::schedule::scheduler: Process 10956 requests memory: 256.00 MiB
10:44:59.177 DEBUG nixie::runtime::migration::migration_plan: Device GlobalDeviceId(0) requires 256.00 MiB of data to migrate in
10:44:59.177 DEBUG nixie::runtime::migration::migration_plan: Created migration task: {"shm -> gpu":{"10956":"0.00 B(0)"},"gpu -> shm":{"10031":"256.00 MiB(2)"}} for 74 us
10:44:59.181 DEBUG nixie::runtime::migration::execution: Data migration completed in 0.004s, largest transfer size = 256.00 MiB, speed = 70.089 GB/s                          
10:44:59.413 DEBUG nixie::runtime::schedule::scheduler: Process 10956 requests memory: 1.50 GiB
10:44:59.413 DEBUG nixie::runtime::migration::migration_plan: Device GlobalDeviceId(1) requires 1.50 GiB of data to migrate in
10:44:59.413 DEBUG nixie::runtime::migration::migration_plan: Created migration task: {"shm -> gpu":{"10956":"0.00 B(0)"},"gpu -> shm":{"10031":"1.50 GiB(12)"}} for 71 us    
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process                                                               
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process                                                               
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process                                                               
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process 
10:45:02.121  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process                                                               
10:45:02.121 DEBUG nixie::runtime::migration::execution: Data migration completed in 2.708s, largest transfer size = 1.50 GiB, speed = 0.595 GB/s
10:45:02.363  INFO nixie::runtime::proc_ctl: ProcessControl [pid=10031] finished                                                                                              
10:45:03.441  INFO nixie::runtime::proc_ctl: ProcessControl [pid=10956] finished                                                                                              
10:45:03.538 DEBUG nixie::runtime::schedule::scheduler: Process 10956 exited                                                                                                  

and the first nixie run process crashes with:

NIXIE-WARN CUDA error from Failed to copy memory from device to host: CUDA_ERROR_INVALID_HANDLE                               
NIXIE-WARN CUDA error from Failed to copy memory from device to host: CUDA_ERROR_INVALID_HANDLE                          
NIXIE-WARN CUDA error from Failed to copy memory from device to host: CUDA_ERROR_INVALID_HANDLE                                                                               
[...]                                                                          
NIXIE-WARN CUDA error from Failed to copy memory from device to host: CUDA_ERROR_INVALID_HANDLE                                                                               
NIXIE-WARN CUDA error from Failed to copy memory from device to host: CUDA_ERROR_INVALID_HANDLE                                                                               
NIXIE-WARN CUDA error from Failed to record CUDA event for d2h copy: CUDA_ERROR_INVALID_HANDLE                                                                                

[XOR-op]

Hi, I couldn't reproduce this issue on my side. Is this problem transient or always happening? If possible, could you provide more details on how to reproduce it? For example how frequently would you encounter it, and what speculation you have over it.

[githubnemo]

Sure, I'll try. This is happening directly at startup when loading the model, consistently on every start.

I simplified the setup a bit, sequentially loading the same model (which doesn't fit more than once) in two separate processes:

nixie daemon --shm 20g -l 'g:0.95'
nixie run -d 0,1 ./bin/llama-server -m ~/.cache/llama.cpp/ggml-org_Qwen3-8B-GGUF_Qwen3-8B-Q4_K_M.gguf -c $((100*1024)) --port 8002 --host 0.0.0.0
nixie run -d 0,1 ./bin/llama-server -m ~/.cache/llama.cpp/ggml-org_Qwen3-8B-GGUF_Qwen3-8B-Q4_K_M.gguf -c $((100*1024)) --port 8003 --host 0.0.0.0

I let them load sequentially. The first process (that needs to yield memory) logs:

0.33.068.725 I srv  llama_server: model loaded                                                                                                                                
0.33.068.729 I srv  llama_server: server is listening on http://0.0.0.0:<port>                                                                                                  
0.33.068.736 I srv  update_slots: all slots are idle                                                                                                                          
NIXIE-WARN CUDA error from Failed to copy memory from device to host: CUDA_ERROR_INVALID_HANDLE     
[... repeated N times ...]
NIXIE-WARN CUDA error from Failed to copy memory from device to host: CUDA_ERROR_INVALID_HANDLE      
NIXIE-WARN CUDA error from Failed to record CUDA event for d2h copy: CUDA_ERROR_INVALID_HANDLE 

the second process (that needs to 'borrow' memory) logs:

[...]
0.06.982.381 I common_init_result: (for bugs during this step try to reproduce them with -fit off, or provide --verbose logs if the bug only occurs with -fit on)
0.23.589.796 W load: control-looking token: 128247 '</s>' was not control-type; this is probably a bug in the model. its type will be overridden
NIXIE-WARN CUDA error from GET_MEM_INFO: CUDA_ERROR_OUT_OF_MEMORY
0.26.749.087 E ggml_backend_cuda_buffer_type_alloc_buffer: allocating 2264.04 MiB on device 1: cudaMalloc failed: out of memory
0.26.749.095 E alloc_tensor_range: failed to allocate CUDA1 buffer of size 2374021120
0.26.790.574 E llama_model_load: error loading model: unable to allocate CUDA1 buffer
0.26.790.595 E llama_model_load_from_file_impl: failed to load model
0.26.790.601 E common_init_from_params: failed to load model '/home/nemo/.cache/llama.cpp/ggml-org_Qwen3-8B-GGUF_Qwen3-8B-Q4_K_M.gguf'
0.26.790.607 E srv    load_model: failed to load model, '/home/nemo/.cache/llama.cpp/ggml-org_Qwen3-8B-GGUF_Qwen3-8B-Q4_K_M.gguf'
0.26.790.636 I srv    operator(): operator(): cleaning up before exit...
0.26.792.445 E srv  llama_server: exiting due to model loading error

and the daemon is logging

18:56:39.193  INFO nixie::logging: Logging initialized. Level=TRACE will be saved in "/tmp/tmp-nixie-daemon-20260906-185639.log"
18:56:39.193  INFO nixie: Starting daemon...
18:56:39.478  INFO nixie::config: Device limit: Global: 0.95(10.45 GiB,10.45 GiB)
18:56:39.478  INFO nixie::runtime::daemon: Shared memory buffer created at /nixie_shm_buffer, size = 20.00 GiB
18:56:39.478  INFO nixie::runtime::schedule::scheduler: Starting scheduler...
18:56:39.479  INFO nixie::runtime::daemon: Daemon started at "/tmp/nixie.sock"
18:56:46.389  INFO nixie::runtime::daemon_server: Client[pid=7239, comm="llama-server"] connected
18:56:46.389 DEBUG nixie::runtime::shm: open_shm(/nixie_ipc-7239-4a14da1f.shm)
18:56:46.390  INFO nixie::runtime::daemon_server: Client[pid=7239] VRAM limit: ["0 (10279 MB)", "1 (10273 MB)"]
18:56:46.390  INFO nixie::runtime::proc_ctl: Listen events from process [pid=7239]
18:56:47.903  INFO nixie::runtime::proc_ctl: ProcessControl [pid=7239] finished
18:57:19.599  INFO nixie::runtime::daemon_server: Client[pid=7348, comm="llama-server"] connected
18:57:19.599 DEBUG nixie::runtime::shm: open_shm(/nixie_ipc-7348-c7b0cd0e.shm)
18:57:19.599  INFO nixie::runtime::daemon_server: Client[pid=7348] VRAM limit: ["1 (10273 MB)", "0 (10279 MB)"]
18:57:19.599  INFO nixie::runtime::proc_ctl: Listen events from process [pid=7348]
18:57:51.378 DEBUG nixie::runtime::schedule::scheduler: Process 7348 requests scheduling
18:57:51.685 DEBUG nixie::runtime::schedule::scheduler: Process 7348 becomes idle
18:57:51.714 DEBUG nixie::runtime::schedule::scheduler: Process 7348 requests scheduling
18:57:52.487 DEBUG nixie::runtime::schedule::scheduler: Process 7348 becomes idle
18:57:53.112 DEBUG nixie::runtime::schedule::scheduler: Process 7348 requests scheduling
18:57:53.564 DEBUG nixie::runtime::schedule::scheduler: Process 7348 becomes idle
18:57:53.632 DEBUG nixie::runtime::schedule::scheduler: Process 7348 requests scheduling
18:57:53.754 DEBUG nixie::runtime::schedule::scheduler: Process 7348 becomes idle
18:58:10.776  INFO nixie::runtime::daemon_server: Client[pid=7580, comm="llama-server"] connected
18:58:10.776 DEBUG nixie::runtime::shm: open_shm(/nixie_ipc-7580-6260d5eb.shm)
18:58:10.776  INFO nixie::runtime::daemon_server: Client[pid=7580] VRAM limit: ["0 (10279 MB)", "1 (10273 MB)"]
18:58:10.776  INFO nixie::runtime::proc_ctl: Listen events from process [pid=7580]
18:58:34.738 DEBUG nixie::runtime::schedule::scheduler: Process 7580 requests scheduling
18:58:34.738 DEBUG nixie::runtime::migration::migration_plan: Device GlobalDeviceId(0) requires 0.00 B of data to migrate in
18:58:34.738 DEBUG nixie::runtime::migration::migration_plan: Device GlobalDeviceId(1) requires 0.00 B of data to migrate in
18:58:34.738 DEBUG nixie::runtime::migration::migration_plan: Created migration task: {"shm -> gpu":{"7580":"0.00 B(0)"}} for 173 us
18:58:34.739 DEBUG nixie::runtime::migration::execution: No migration needed
18:58:35.046 DEBUG nixie::runtime::schedule::scheduler: Process 7580 becomes idle
18:58:35.133 DEBUG nixie::runtime::schedule::scheduler: Process 7580 requests memory: 256.00 MiB
18:58:35.133 DEBUG nixie::runtime::migration::migration_plan: Device GlobalDeviceId(0) requires 256.00 MiB of data to migrate in
18:58:35.133 DEBUG nixie::runtime::migration::migration_plan: Created migration task: {"gpu -> shm":{"7348":"256.00 MiB(2)"},"shm -> gpu":{"7580":"0.00 B(0)"}} for 80 us
18:58:35.138 DEBUG nixie::runtime::migration::execution: Data migration completed in 0.004s, largest transfer size = 256.00 MiB, speed = 64.337 GB/s
18:58:35.365 DEBUG nixie::runtime::schedule::scheduler: Process 7580 requests memory: 896.00 MiB
18:58:35.365 DEBUG nixie::runtime::migration::migration_plan: Device GlobalDeviceId(1) requires 896.00 MiB of data to migrate in
18:58:35.365 DEBUG nixie::runtime::migration::migration_plan: Created migration task: {"shm -> gpu":{"7580":"0.00 B(0)"},"gpu -> shm":{"7348":"896.00 MiB(7)"}} for 60 us
18:58:37.507  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
18:58:37.507  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
18:58:37.507  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
18:58:37.507  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
18:58:37.507  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
18:58:37.507  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
18:58:37.507  WARN nixie::runtime::migration::execution: Failed to complete D2H migration RPC to source process
18:58:37.507 DEBUG nixie::runtime::migration::execution: Data migration completed in 2.142s, largest transfer size = 896.00 MiB, speed = 0.439 GB/s
18:58:38.618  INFO nixie::runtime::proc_ctl: ProcessControl [pid=7580] finished
18:58:38.624 DEBUG nixie::runtime::schedule::scheduler: Process 7580 exited

I've attached the mentioned trace log.

This is also happening for nixie daemon --shm 20g --hostmem 40g -l 'g:0.90'

There is no useful message in dmesg.

Disclaimer: the kernel was updated to 7.0 in the meantime and I was using a kernel with disabled huge pages before but since the issue is still there I would rule these out as the source of the error.

tmp-nixie-daemon-20260906-185639.log

Edit: I tested this with only device 0 and the result is the same, although the first process isn't complaining about failing d2h copy, only the nixie daemon is.

[XOR-op]

Sadly I still cannot reproduce this bug, while I identified some other bugs and fixed them. FYI this is how I test it:

nixie daemon --shm 20g -l 'g:0.95' 
nixie run -d 0,1 ./build/bin/llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_XL -c 262144 --port 8002 --host 0.0.0.0
nixie run -d 0,1 ./build/bin/llama-server -hf unsloth/Qwen3.8-27B-GGUF:UD-Q6_K_XL -c 262144 --port 8003 --host 0.0.0.0

Then curl to first server, then second server. llama.cpp is in commit 465e49b9c. My rig spec is 2x5090 32G, 128GB DRAM.

I'm not sure if the bugs I fixed are relevant here. Given the error is CUDA_ERROR_INVALID_HANDLE, it's likely to be invalid CUDA stream issue in the memcpy; however I cannot either reproduce it or let LLM to give me useful insights. And this won't be a problem if you only use GPU 0. So I suspect your device0-only issue is not from here. Also you might try reboot the machine after the error? That may sound dumb, but llama.cpp and nixie sometimes cannot handle cleanup well, so perhaps some stale states are affecting the result.

If possible, could you update both to latest version and send me the GPU0-only log when you saw the error? I feel that may help me debug this.