[Bug]: With ROCm and certain HF models that require 'trust-remote-code', you get VLLM_RPC_TIMEOUT and failure to finish loading.

[philtimmes]

Your current environment

The output of `python collect_env.py`

user1@dev1:~/vllm$ python collect_env.py
WARNING 12-16 05:24:35 rocm.py:31] `fork` method is not supported by ROCm. VLLM_WORKER_MULTIPROC_METHOD is overridden to `spawn` instead.
Collecting environment information...
PyTorch version: 2.6.0.dev20241211+rocm6.1
Is debug build: False
CUDA used to build PyTorch: N/A
ROCM used to build PyTorch: 6.1.40091-a8dbc0c19

OS: Ubuntu 22.04.5 LTS (x86_64)
GCC version: (conda-forge gcc 12.1.0-17) 12.1.0
Clang version: Could not collect
CMake version: version 3.31.1
Libc version: glibc-2.35

Python version: 3.10.16 | packaged by conda-forge | (main, Dec  5 2024, 14:16:10) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-5.15.0-97-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.7.64
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: AMD Instinct MI250X/MI250 (gfx90a:sramecc+:xnack-)
Nvidia driver version: Could not collect
cuDNN version: Could not collect
HIP runtime version: 6.1.40091
MIOpen runtime version: 3.1.0
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             32
On-line CPU(s) list:                0-31
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 73F3 16-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 1
Core(s) per socket:                 16
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        4036.6211
CPU min MHz:                        1500.0000
BogoMIPS:                           6986.65
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf rapl pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr rdpru wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif v_spec_ctrl umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca fsrm
Virtualization:                     AMD-V
L1d cache:                          1 MiB (32 instances)
L1i cache:                          1 MiB (32 instances)
L2 cache:                           16 MiB (32 instances)
L3 cache:                           512 MiB (16 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-15
NUMA node1 CPU(s):                  16-31
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec rstack overflow: Mitigation; safe RET
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP disabled, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==11.525.150
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] onnxruntime==1.18.0
[pip3] open-clip-torch==2.24.0
[pip3] pytorch-lightning==2.3.0
[pip3] pytorch-triton-rocm==3.2.0+git35c6c7c6
[pip3] pyzmq==25.1.2
[pip3] torch==2.6.0.dev20241211+rocm6.1
[pip3] torchaudio==2.5.0.dev20241212+rocm6.1
[pip3] torchdiffeq==0.2.4
[pip3] torchmetrics==1.4.0.post0
[pip3] torchsde==0.2.6
[pip3] torchvision==0.22.0.dev20241212+rocm6.1
[pip3] transformers==4.47.0
[pip3] triton==3.1.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pytorch-triton-rocm       3.2.0+git35c6c7c6          pypi_0    pypi
[conda] torch                     2.6.0.dev20241211+rocm6.1          pypi_0    pypi
[conda] torchaudio                2.5.0.dev20241212+rocm6.1          pypi_0    pypi
[conda] torchvision               0.22.0.dev20241212+rocm6.1          pypi_0    pypi
[conda] transformers              4.47.0                   pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi
ROCM Version: 6.3.42131-fa1d09cbd
Neuron SDK Version: N/A
vLLM Version: 0.6.4.post2.dev378+g69ba344d
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
============================ ROCm System Management Interface ============================
================================ Weight between two GPUs =================================
       GPU0         GPU1         GPU2         GPU3         GPU4         GPU5         GPU6         GPU7
GPU0   0            15           15           30           30           30           15           30
GPU1   15           0            30           15           30           15           30           45
GPU2   15           30           0            15           15           30           30           30
GPU3   30           15           15           0            30           45           30           15
GPU4   30           30           15           30           0            15           15           30
GPU5   30           15           30           45           15           0            30           15
GPU6   15           30           30           30           15           30           0            15
GPU7   30           45           30           15           30           15           15           0

================================= Hops between two GPUs ==================================
       GPU0         GPU1         GPU2         GPU3         GPU4         GPU5         GPU6         GPU7
GPU0   0            1            1            1            1            1            1            1
GPU1   1            0            1            1            1            1            1            1
GPU2   1            1            0            1            1            1            1            1
GPU3   1            1            1            0            1            1            1            1
GPU4   1            1            1            1            0            1            1            1
GPU5   1            1            1            1            1            0            1            1
GPU6   1            1            1            1            1            1            0            1
GPU7   1            1            1            1            1            1            1            0

=============================== Link Type between two GPUs ===============================
       GPU0         GPU1         GPU2         GPU3         GPU4         GPU5         GPU6         GPU7
GPU0   0            XGMI         XGMI         XGMI         XGMI         XGMI         XGMI         XGMI
GPU1   XGMI         0            XGMI         XGMI         XGMI         XGMI         XGMI         XGMI
GPU2   XGMI         XGMI         0            XGMI         XGMI         XGMI         XGMI         XGMI
GPU3   XGMI         XGMI         XGMI         0            XGMI         XGMI         XGMI         XGMI
GPU4   XGMI         XGMI         XGMI         XGMI         0            XGMI         XGMI         XGMI
GPU5   XGMI         XGMI         XGMI         XGMI         XGMI         0            XGMI         XGMI
GPU6   XGMI         XGMI         XGMI         XGMI         XGMI         XGMI         0            XGMI
GPU7   XGMI         XGMI         XGMI         XGMI         XGMI         XGMI         XGMI         0

======================================= Numa Nodes =======================================
GPU[0]          : (Topology) Numa Node: 0
GPU[0]          : (Topology) Numa Affinity: 0
GPU[1]          : (Topology) Numa Node: 0
GPU[1]          : (Topology) Numa Affinity: 0
GPU[2]          : (Topology) Numa Node: 0
GPU[2]          : (Topology) Numa Affinity: 0
GPU[3]          : (Topology) Numa Node: 0
GPU[3]          : (Topology) Numa Affinity: 0
GPU[4]          : (Topology) Numa Node: 1
GPU[4]          : (Topology) Numa Affinity: 1
GPU[5]          : (Topology) Numa Node: 1
GPU[5]          : (Topology) Numa Affinity: 1
GPU[6]          : (Topology) Numa Node: 1
GPU[6]          : (Topology) Numa Affinity: 1
GPU[7]          : (Topology) Numa Node: 1
GPU[7]          : (Topology) Numa Affinity: 1
================================== End of ROCm SMI Log ===================================

LD_LIBRARY_PATH=/home/user1/.local/lib/python3.10/site-packages/cv2/../../lib64:/opt/rocm/lib:/opt/nvidia/cuda-11.7/lib64
VLLM_WORKER_MULTIPROC_METHOD=spawn
CUDA_MODULE_LOADING=LAZY

Model Input Dumps

(VllmWorkerProcess pid=5932) /opt/miniconda3/envs/vllm/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:524: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
(VllmWorkerProcess pid=5932)   warnings.warn(
/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:524: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
  warnings.warn(
(VllmWorkerProcess pid=5934) /opt/miniconda3/envs/vllm/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:524: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
(VllmWorkerProcess pid=5934)   warnings.warn(
(VllmWorkerProcess pid=5933) /opt/miniconda3/envs/vllm/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:524: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
(VllmWorkerProcess pid=5933)   warnings.warn(
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<MQLLMEngineClient.run_output_handler_loop() done, defined at /opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/engine/multiprocessing/client.py:178> exception=ZMQError('Operation not supported')>
Traceback (most recent call last):
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/engine/multiprocessing/client.py", line 184, in run_output_handler_loop
    while await self.output_socket.poll(timeout=VLLM_RPC_TIMEOUT
  File "/home/realai/.local/lib/python3.10/site-packages/zmq/_future.py", line 382, in poll
    raise _zmq.ZMQError(_zmq.ENOTSUP)
zmq.error.ZMQError: Operation not supported
Task exception was never retrieved
future: <Task finished name='Task-3' coro=<MQLLMEngineClient.run_output_handler_loop() done, defined at /opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/engine/multiprocessing/client.py:178> exception=ZMQError('Operation not supported')>
Traceback (most recent call last):
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/engine/multiprocessing/client.py", line 184, in run_output_handler_loop
    while await self.output_socket.poll(timeout=VLLM_RPC_TIMEOUT
  File "/home/realai/.local/lib/python3.10/site-packages/zmq/_future.py", line 382, in poll
    raise _zmq.ZMQError(_zmq.ENOTSUP)
zmq.error.ZMQError: Operation not supported
Traceback (most recent call last):
  File "/opt/miniconda3/envs/vllm/bin/vllm", line 33, in <module>
    sys.exit(load_entry_point('vllm==0.6.4.post2.dev378+g69ba344d.rocm634', 'console_scripts', 'vllm')())
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/scripts.py", line 201, in main
    args.dispatch_function(args)
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/scripts.py", line 42, in serve
    uvloop.run(run_server(args))
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 649, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/opt/miniconda3/envs/vllm/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 116, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/opt/miniconda3/envs/vllm/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 213, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
(vllm) realai@c1:~$ /opt/miniconda3/envs/vllm/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 16 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/opt/miniconda3/envs/vllm/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

🐛 Describe the bug

As stated above:
With ROCm and certain HF models that require 'trust-remote-code', you get VLLM_RPC_TIMEOUT and failure to finish loading.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[hongxiayang]

@philtimmes Can you increase the timeout and try again? VLLM_RPC_TIMEOUT=30000 or more

[philtimmes]

WARNING 12-17 20:01:41 rocm.py:31] `fork` method is not supported by ROCm. VLLM_WORKER_MULTIPROC_METHOD is overridden to `spawn` instead.
INFO 12-17 20:01:42 api_server.py:625] vLLM API server version 0.6.4.post2.dev266+g7be15d93
INFO 12-17 20:01:42 api_server.py:626] args: Namespace(subparser='serve', model_tag='openbmb/MiniCPM-V-2_6', config='', host='127.0.0.1', port=5678, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='openbmb/MiniCPM-V-2_6', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=None, guided_decoding_backend='xgrammar', distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=4, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=16, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.9, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=True, max_seq_len_to_capture=131072, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=30, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=False, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=['RealTruth.ai'], qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=None, worker_cls='auto', disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False, dispatch_function=<function serve at 0x7f03db2dd120>)
INFO 12-17 20:01:42 api_server.py:197] Started engine process with PID 229245
INFO 12-17 20:01:49 config.py:404] This model supports multiple tasks: {'generate', 'embedding'}. Defaulting to 'generate'.
INFO 12-17 20:01:49 config.py:1039] Defaulting to use mp for distributed inference
INFO 12-17 20:01:49 config.py:1067] Disabled the custom all-reduce kernel because it is not supported on AMD GPUs.
WARNING 12-17 20:01:49 config.py:532] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 12-17 20:01:52 config.py:404] This model supports multiple tasks: {'generate', 'embedding'}. Defaulting to 'generate'.
INFO 12-17 20:01:52 config.py:1039] Defaulting to use mp for distributed inference
INFO 12-17 20:01:52 config.py:1067] Disabled the custom all-reduce kernel because it is not supported on AMD GPUs.
WARNING 12-17 20:01:52 config.py:532] To see benefits of async output processing, enable CUDA graph. Since, enforce-eager is enabled, async output processor cannot be used
INFO 12-17 20:01:52 llm_engine.py:249] Initializing an LLM engine (v0.6.4.post2.dev266+g7be15d93) with config: model='openbmb/MiniCPM-V-2_6', speculative_config=None, tokenizer='openbmb/MiniCPM-V-2_6', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=True, quantization=None, enforce_eager=True, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=RealTruth.ai, num_scheduler_steps=30, chunked_prefill_enabled=False multi_step_stream_outputs=True, enable_prefix_caching=False, use_async_output_proc=False, use_cached_outputs=True, mm_processor_kwargs=None, pooler_config=None,compilation_config=CompilationConfig(level=0, backend='', custom_ops=[], splitting_ops=['vllm.unified_attention', 'vllm.unified_attention_with_output'], use_inductor=True, inductor_specialize_for_cudagraph_no_more_than=None, inductor_compile_sizes=[], inductor_compile_config={}, inductor_passes={}, use_cudagraph=False, cudagraph_num_of_warmups=0, cudagraph_capture_sizes=None, cudagraph_copy_inputs=False, pass_config=PassConfig(dump_graph_stages=[], dump_graph_dir=PosixPath('.'), enable_fusion=True, enable_reshape=True), compile_sizes=[], capture_sizes=[], enabled_custom_ops=Counter(), disabled_custom_ops=Counter(), compilation_time=0.0, static_forward_context={})
WARNING 12-17 20:01:53 multiproc_gpu_executor.py:53] Reducing Torch parallelism from 32 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed.
INFO 12-17 20:01:53 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 12-17 20:01:56 selector.py:134] Using ROCmFlashAttention backend.
WARNING 12-17 20:01:56 registry.py:245] `mm_limits` has already been set for model=openbmb/MiniCPM-V-2_6, and will be overwritten by the new values.
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m INFO 12-17 20:01:58 selector.py:134] Using ROCmFlashAttention backend.
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m INFO 12-17 20:01:58 selector.py:134] Using ROCmFlashAttention backend.
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m INFO 12-17 20:01:58 selector.py:134] Using ROCmFlashAttention backend.
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m WARNING 12-17 20:01:58 registry.py:245] `mm_limits` has already been set for model=openbmb/MiniCPM-V-2_6, and will be overwritten by the new values.
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m INFO 12-17 20:01:58 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m WARNING 12-17 20:01:58 registry.py:245] `mm_limits` has already been set for model=openbmb/MiniCPM-V-2_6, and will be overwritten by the new values.
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m INFO 12-17 20:01:58 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m WARNING 12-17 20:01:58 registry.py:245] `mm_limits` has already been set for model=openbmb/MiniCPM-V-2_6, and will be overwritten by the new values.
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m INFO 12-17 20:01:58 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
INFO 12-17 20:01:59 utils.py:988] Found nccl from library librccl.so.1
INFO 12-17 20:01:59 pynccl.py:69] vLLM is using nccl==2.18.6
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m INFO 12-17 20:01:59 utils.py:988] Found nccl from library librccl.so.1
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m INFO 12-17 20:01:59 utils.py:988] Found nccl from library librccl.so.1
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m INFO 12-17 20:01:59 utils.py:988] Found nccl from library librccl.so.1
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m INFO 12-17 20:01:59 pynccl.py:69] vLLM is using nccl==2.18.6
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m INFO 12-17 20:01:59 pynccl.py:69] vLLM is using nccl==2.18.6
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m INFO 12-17 20:01:59 pynccl.py:69] vLLM is using nccl==2.18.6
INFO 12-17 20:01:59 shm_broadcast.py:236] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer=<vllm.distributed.device_communicators.shm_broadcast.ShmRingBuffer object at 0x7fe6c6746c20>, local_subscribe_port=46433, remote_subscribe_port=None)
INFO 12-17 20:01:59 model_runner.py:1089] Starting to load model openbmb/MiniCPM-V-2_6...
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m INFO 12-17 20:01:59 model_runner.py:1089] Starting to load model openbmb/MiniCPM-V-2_6...
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m INFO 12-17 20:01:59 model_runner.py:1089] Starting to load model openbmb/MiniCPM-V-2_6...
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m INFO 12-17 20:01:59 model_runner.py:1089] Starting to load model openbmb/MiniCPM-V-2_6...
INFO 12-17 20:01:59 selector.py:134] Using ROCmFlashAttention backend.
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m INFO 12-17 20:01:59 selector.py:134] Using ROCmFlashAttention backend.
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m INFO 12-17 20:01:59 selector.py:134] Using ROCmFlashAttention backend.
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m INFO 12-17 20:01:59 selector.py:134] Using ROCmFlashAttention backend.
INFO 12-17 20:01:59 weight_utils.py:243] Using model weights format ['*.safetensors']
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m INFO 12-17 20:02:00 weight_utils.py:243] Using model weights format ['*.safetensors']
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m INFO 12-17 20:02:00 weight_utils.py:243] Using model weights format ['*.safetensors']
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m INFO 12-17 20:02:00 weight_utils.py:243] Using model weights format ['*.safetensors']

Loading safetensors checkpoint shards:   0% Completed | 0/4 [00:00<?, ?it/s]

Loading safetensors checkpoint shards:  25% Completed | 1/4 [00:00<00:00,  5.43it/s]

Loading safetensors checkpoint shards:  50% Completed | 2/4 [00:00<00:00,  3.71it/s]

Loading safetensors checkpoint shards:  75% Completed | 3/4 [00:00<00:00,  3.30it/s]

Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00,  2.86it/s]

Loading safetensors checkpoint shards: 100% Completed | 4/4 [00:01<00:00,  3.13it/s]

INFO 12-17 20:02:01 model_runner.py:1094] Loading model weights took 3.9207 GB
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m INFO 12-17 20:02:01 model_runner.py:1094] Loading model weights took 3.9207 GB
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m INFO 12-17 20:02:02 model_runner.py:1094] Loading model weights took 3.9207 GB
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m INFO 12-17 20:02:02 model_runner.py:1094] Loading model weights took 3.9207 GB
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m /opt/miniconda3/envs/vllm/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:524: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
�[1;36m(VllmWorkerProcess pid=229456)�[0;0m   warnings.warn(
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m /opt/miniconda3/envs/vllm/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:524: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
�[1;36m(VllmWorkerProcess pid=229455)�[0;0m   warnings.warn(
/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:524: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
  warnings.warn(
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m /opt/miniconda3/envs/vllm/lib/python3.10/site-packages/transformers/models/auto/image_processing_auto.py:524: FutureWarning: The image_processor_class argument is deprecated and will be removed in v4.42. Please use `slow_image_processor_class`, or `fast_image_processor_class` instead
�[1;36m(VllmWorkerProcess pid=229457)�[0;0m   warnings.warn(
Traceback (most recent call last):
  File "/opt/miniconda3/envs/vllm/bin/vllm", line 33, in <module>
    sys.exit(load_entry_point('vllm==0.6.4.post2.dev396+gf9ecbb18.rocm634', 'console_scripts', 'vllm')())
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/scripts.py", line 201, in main
    args.dispatch_function(args)
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/scripts.py", line 42, in serve
    uvloop.run(run_server(args))
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/uvloop/__init__.py", line 82, in run
    return loop.run_until_complete(wrapper())
  File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/uvloop/__init__.py", line 61, in wrapper
    return await main
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 649, in run_server
    async with build_async_engine_client(args) as engine_client:
  File "/opt/miniconda3/envs/vllm/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 116, in build_async_engine_client
    async with build_async_engine_client_from_engine_args(
  File "/opt/miniconda3/envs/vllm/lib/python3.10/contextlib.py", line 199, in __aenter__
    return await anext(self.gen)
  File "/opt/miniconda3/envs/vllm/lib/python3.10/site-packages/vllm-0.6.4.post2.dev266+g7be15d93.rocm634-py3.10-linux-x86_64.egg/vllm/entrypoints/openai/api_server.py", line 213, in build_async_engine_client_from_engine_args
    raise RuntimeError(
RuntimeError: Engine process failed to start. See stack trace for the root cause.
/opt/miniconda3/envs/vllm/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 16 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/opt/miniconda3/envs/vllm/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '