[Bug]: [0.6.5] Qwen2-VL LoRA Adapters does not take effect.

[Bingogogogogo]

Your current environment

The output of `python collect_env.py`

Collecting environment information... PyTorch version: 2.5.1+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch: N/A

GCC version: (GCC) 11.2.1 20210728 (Red Hat 11.2.1-1)
Clang version: 16.0.6 (Red Hat 16.0.6-2.module+el8.8.0+557+454507bd)
CMake version: version 3.26.5
Libc version: glibc-2.28

Python version: 3.12.5 (main, Sep 24 2024, 12:12:08)
Python platform: Linux-5.4.241-1-tlinux4-0017.7-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 12.4.131
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA L40
Nvidia driver version: 535.161.08
cuDNN version: Probably one of the following:
/usr/lib64/libcudnn.so.9.1.1
/usr/lib64/libcudnn_adv.so.9.1.1
/usr/lib64/libcudnn_cnn.so.9.1.1
/usr/lib64/libcudnn_engines_precompiled.so.9.1.1
/usr/lib64/libcudnn_engines_runtime_compiled.so.9.1.1
/usr/lib64/libcudnn_graph.so.9.1.1
/usr/lib64/libcudnn_heuristic.so.9.1.1
/usr/lib64/libcudnn_ops.so.9.1.1
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
CPU(s): 384
On-line CPU(s) list: 0-383
Thread(s) per core: 2
Core(s) per socket: 96
Socket(s): 2
NUMA node(s): 2
Vendor ID: AuthenticAMD
CPU family: 25
Model: 17
Model name: AMD EPYC 9K84 96-Core Processor
Stepping: 0
CPU MHz: 2600.024
BogoMIPS: 5200.04
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 32K
L1i cache: 32K
L2 cache: 1024K
L3 cache: 32768K
NUMA node0 CPU(s): 0-191
NUMA node1 CPU(s): 192-383
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid amd_dcm tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw topoext perfctr_core invpcid_single ibpb vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 avx512_bf16 clzero xsaveerptr wbnoinvd arat avx512vbmi umip avx512_vbmi2 vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid fsrm

Versions of relevant libraries:
[pip3] flamingo-pytorch==0.1.2
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.6.4.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.1.105
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] open_clip_torch==2.28.0
[pip3] optree==0.13.0
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1+cu121
[pip3] torchaudio==2.5.0a0+56bc006
[pip3] torchvision==0.20.1
[pip3] transformers==4.46.1
[pip3] transformers-stream-generator==0.0.4
[pip3] triton==3.1.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.5
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X 192-383 1 N/A

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

LD_LIBRARY_PATH=/usr/local/lib64/python3.12/site-packages/cv2/../../lib64:/opt/rh/gcc-toolset-11/root/usr/lib64:/opt/rh/gcc-toolset-11/root/usr/lib:/opt/rh/gcc-toolset-11/root/usr/lib64/dyninst:/opt/rh/gcc-toolset-11/root/usr/lib/dyninst
CUDA_MODULE_LOADING=LAZY

Model Input Dumps

No response

🐛 Describe the bug

Hi, there. I have finetuned Qwen2-VL-7B-Instruct with LLaMA-Factory in two different tasks and got two lora weights(named lora1 and lora2). However, when I start an server with base model and these two lora adapters, just follow the document(https://docs.vllm.ai/en/latest/usage/lora.html), I found that no matter which lora model I request for, the response just as same as the base model.

As I print the details in https://github.com/vllm-project/vllm/blob/29c748930e0d35a98351a8cf8a093fba4b758114/vllm/lora/models.py#L368，

I found register modules are named with "language_model.model.layers.XXX" (maybe same as base model), however, the loaded weights of lora adapters are named with "model.layers.XXX"(processed by function parse_fine_tuned_lora_name). The difference makes module_lora=None and the foward process does not pass through the lora module.

I have not met with the same problem with 0.6.4post1, and I wonder if I misunderstand the usage in new version.

python3 -m vllm.entrypoints.openai.api_server --model $MODEL --served-model-name $SERVED_MODEL_NAME --enable-lora --lora-modules $LORA_MODULES --host 0.0.0.0 --port 12345 --trust-remote-code --max_model_len 1200 --dtype auto --limit-mm-per-prompt image=4,video=2 --chat-template-content-format openai --disable-frontend-multiprocessing

{ "object": "list", "data": [ { "id": "Qwen2-VL-7B-Instruct", "object": "model", "created": 1734852933, "owned_by": "vllm", "root": "base_model_path", "parent": null, "max_model_len": 1200, "permission": [ { "id": "modelperm-8b3105807626418d8377fabb95e34611", "object": "model_permission", "created": 1734852933, "allow_create_engine": false, "allow_sampling": true, "allow_logprobs": true, "allow_search_indices": false, "allow_view": true, "allow_fine_tuning": false, "organization": "*", "group": null, "is_blocking": false } ] }, { "id": "lora1", "object": "model", "created": 1734852933, "owned_by": "vllm", "root": "lora1_path", "parent": "Qwen2-VL-7B-Instruct", "max_model_len": null, "permission": [ { "id": "modelperm-179fdf3870524fbbb45474e1f0a95767", "object": "model_permission", "created": 1734852933, "allow_create_engine": false, "allow_sampling": true, "allow_logprobs": true, "allow_search_indices": false, "allow_view": true, "allow_fine_tuning": false, "organization": "*", "group": null, "is_blocking": false } ] }, { "id": "lora2", "object": "model", "created": 1734852933, "owned_by": "vllm", "root": "lora2_path", "parent": "Qwen2-VL-7B-Instruct", "max_model_len": null, "permission": [ { "id": "modelperm-16e0e3cc984e437eb829a54c5c2af7f7", "object": "model_permission", "created": 1734852933, "allow_create_engine": false, "allow_sampling": true, "allow_logprobs": true, "allow_search_indices": false, "allow_view": true, "allow_fine_tuning": false, "organization": "*", "group": null, "is_blocking": false } ] } ] }

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[jeejeelee]

I will try to reproduce your issue first.

[jeejeelee]

This is a bug that we are working on to fix.

[Bingogogogogo]

This is a bug that we are working on to fix.

thanks

[jeejeelee]

Hi, I've submitted a PR to fix this. Could you please verify if it addresses your issue, see: #11430

[Bingogogogogo]

Hi, I've submitted a PR to fix this. Could you please verify if it addresses your issue, see: #11430

How can i verify this PR, reinstall vllm from source？

[jeejeelee]

Hi, I've submitted a PR to fix this. Could you please verify if it addresses your issue, see: #11430

How can i verify this PR, reinstall vllm from source？

Clone my branch, then build from source