Start Qwen-27B vllm serving failed on B60

[HoppeDeng]

The system has 32GB ram, the vllm cmd is below:
export VLLM_OFFLOAD_WEIGHTS_BEFORE_QUANT=0
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1
VLLM_WORKER_MULTIPROC_METHOD=spawn
vllm serve
--model /llm/models/Qwen3.6-27B
--served-model-name Qwen3.6-27B
--dtype=float16
--enforce-eager
--port 8000
--host 0.0.0.0
--trust-remote-code
--disable-sliding-window
--gpu-memory-util=0.8
--max-num-batched-tokens=8192
--max-model-len=131072
--block-size 64
--quantization sym_int4
-tp=1

The error log is
(EngineCore pid=291) Process EngineCore: 16:12:45 [62/152]
(EngineCore pid=291) Traceback (most recent call last):
(EngineCore pid=291) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=291) self.run()
(EngineCore pid=291) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=291) self._target(*self._args, **self._kwargs)
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1145, in run_engine_core
(EngineCore pid=291) raise e
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1115, in run_engine_core
(EngineCore pid=291) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=291) return func(*args, **kwargs)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 881, in init
(EngineCore pid=291) super().init(
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 118, in init
(EngineCore pid=291) self.model_executor = executor_class(vllm_config)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=291) return func(*args, **kwargs)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/abstract.py", line 109, in init
(EngineCore pid=291) self._init_executor()
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/executor/uniproc_executor.py", line 60, in _init_executor
(EngineCore pid=291) self.driver_worker.init_device()
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/worker_base.py", line 317, in init_device
(EngineCore pid=291) self.worker.init_device() # type: ignore
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/vllm/v1/worker/xpu_worker.py", line 90, in init_device
(EngineCore pid=291) torch.distributed.all_reduce(torch.zeros(1).xpu())
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
(EngineCore pid=291) return func(*args, **kwargs)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) File "/opt/venv/lib/python3.12/site-packages/torch/distributed/distributed_c10d.py", line 3068, in all_reduce
(EngineCore pid=291) work = group.allreduce([tensor], opts)
(EngineCore pid=291) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=291) RuntimeError: level_zero backend failed with error: 39 (UR_RESULT_ERROR_OUT_OF_DEVICE_MEMORY)

[365614269]

Also got an error server trace on intel/llm-scaler-vllm:0.21.0-b3.1, but it complains about a lack of memory instead of level zero backend. Note that I had a normal and working startup with --max-model-len=16384 and --gpu-memory-util=0.9, so your --max-model-len seems a bit too aggressive. I've also tried a few other --max-model-lens, and 16384 is my largest working value.

Container startup:

docker run -it \
    --privileged \
    --net=host \
    --device=/dev/dri \
    --name mark-b3.1-container \
    --device=/dev/dri \
    --shm-size=16g \
    -v /home/intel/weights:/models/:ro \
    -v /home/intel/mark/llm-scaler:/llm-scaler \
    -v /home/intel/mark/:/mark \
    -e no_proxy=localhost,127.0.0.1 \
    -e http_proxy=http://proxy.iil.intel.com:911 \
    -e https_proxy=http://proxy.iil.intel.com:911 \
    -m 32g \
    --entrypoint bash \
    intel/llm-scaler-vllm:0.21.0-b3.1

Server startup command: Exactly the same as yours, as well as the version with tweaks I mentioned earlier.

Server trace with your exact command:

(EngineCore pid=548) INFO 08-17 06:40:57 [gpu_worker.py:462] Available KV cache memory: -0.83 GiB
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141] EngineCore failed to start.
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141] Traceback (most recent call last):
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1115, in run_engine_core
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]   File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]     return func(*args, **kwargs)
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 881, in __init__
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]     super().__init__(
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]   File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]     return func(*args, **kwargs)
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 264, in _initialize_kv_caches
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]     kv_cache_configs = get_kv_cache_configs(
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]                        ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/core/kv_cache_utils.py", line 2015, in get_kv_cache_configs
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]     _check_enough_kv_cache_memory(
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/core/kv_cache_utils.py", line 696, in _check_enough_kv_cache_memory
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141]     raise ValueError(
(EngineCore pid=548) ERROR 08-17 06:40:57 [core.py:1141] ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.
(EngineCore pid=548) Process EngineCore:
(EngineCore pid=548) Traceback (most recent call last):
(EngineCore pid=548)   File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=548)     self.run()
(EngineCore pid=548)   File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=548)     self._target(*self._args, **self._kwargs)
(EngineCore pid=548)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1145, in run_engine_core
(EngineCore pid=548)     raise e
(EngineCore pid=548)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 1115, in run_engine_core
(EngineCore pid=548)     engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=548)                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548)   File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=548)     return func(*args, **kwargs)
(EngineCore pid=548)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 881, in __init__
(EngineCore pid=548)     super().__init__(
(EngineCore pid=548)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 128, in __init__
(EngineCore pid=548)     kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=548)                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548)   File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=548)     return func(*args, **kwargs)
(EngineCore pid=548)            ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core.py", line 264, in _initialize_kv_caches
(EngineCore pid=548)     kv_cache_configs = get_kv_cache_configs(
(EngineCore pid=548)                        ^^^^^^^^^^^^^^^^^^^^^
(EngineCore pid=548)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/core/kv_cache_utils.py", line 2015, in get_kv_cache_configs
(EngineCore pid=548)     _check_enough_kv_cache_memory(
(EngineCore pid=548)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/core/kv_cache_utils.py", line 696, in _check_enough_kv_cache_memory
(EngineCore pid=548)     raise ValueError(
(EngineCore pid=548) ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` when initializing the engine. See https://docs.vllm.ai/en/latest/configuration/conserving_memory/ for more details.
(APIServer pid=28) Traceback (most recent call last):
(APIServer pid=28)   File "/opt/venv/bin/vllm", line 6, in <module>
(APIServer pid=28)     sys.exit(main())
(APIServer pid=28)              ^^^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=28)     args.dispatch_function(args)
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=28)     uvloop.run(run_server(args))
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 96, in run
(APIServer pid=28)     return __asyncio.run(
(APIServer pid=28)            ^^^^^^^^^^^^^^
(APIServer pid=28)   File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run
(APIServer pid=28)     return runner.run(main)
(APIServer pid=28)            ^^^^^^^^^^^^^^^^
(APIServer pid=28)   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
(APIServer pid=28)     return self._loop.run_until_complete(task)
(APIServer pid=28)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=28)   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/uvloop/__init__.py", line 48, in wrapper
(APIServer pid=28)     return await main
(APIServer pid=28)            ^^^^^^^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 693, in run_server
(APIServer pid=28)     await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 707, in run_server_worker
(APIServer pid=28)     async with build_async_engine_client(
(APIServer pid=28)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=28)     return await anext(self.gen)
(APIServer pid=28)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=28)     async with build_async_engine_client_from_engine_args(
(APIServer pid=28)   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
(APIServer pid=28)     return await anext(self.gen)
(APIServer pid=28)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=28)     async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=28)                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=28)     return cls(
(APIServer pid=28)            ^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/async_llm.py", line 146, in __init__
(APIServer pid=28)     self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=28)                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=28)     return func(*args, **kwargs)
(APIServer pid=28)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=28)     return AsyncMPClient(*client_args)
(APIServer pid=28)            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=28)     return func(*args, **kwargs)
(APIServer pid=28)            ^^^^^^^^^^^^^^^^^^^^^
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 907, in __init__
(APIServer pid=28)     super().__init__(
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/core_client.py", line 535, in __init__
(APIServer pid=28)     with launch_core_engines(
(APIServer pid=28)   File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__
(APIServer pid=28)     next(self.gen)
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1128, in launch_core_engines
(APIServer pid=28)     wait_for_engine_startup(
(APIServer pid=28)   File "/opt/venv/lib/python3.12/site-packages/vllm/v1/engine/utils.py", line 1187, in wait_for_engine_startup
(APIServer pid=28)     raise RuntimeError(
(APIServer pid=28) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}