Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: disaggregated prefilling hangs when TP=2 #11247

Open
1 task done
Louis-99 opened this issue Dec 17, 2024 · 0 comments
Open
1 task done

[Bug]: disaggregated prefilling hangs when TP=2 #11247

Louis-99 opened this issue Dec 17, 2024 · 0 comments
Labels
bug Something isn't working

Comments

@Louis-99
Copy link

Your current environment

The output of `python collect_env.py` 
Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: Could not collect
CMake version: version 3.31.1
Libc version: glibc-2.31

Python version: 3.11.11 | packaged by conda-forge | (main, Dec  5 2024, 14:17:24) [GCC 13.3.0] (64-bit runtime)
Python platform: Linux-5.10.0-33-cloud-amd64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.6.85
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: 
GPU 0: NVIDIA A100-SXM4-80GB
GPU 1: NVIDIA A100-SXM4-80GB
GPU 2: NVIDIA A100-SXM4-80GB
GPU 3: NVIDIA A100-SXM4-80GB
GPU 4: NVIDIA A100-SXM4-80GB
GPU 5: NVIDIA A100-SXM4-80GB
GPU 6: NVIDIA A100-SXM4-80GB
GPU 7: NVIDIA A100-SXM4-80GB

Nvidia driver version: 550.90.07
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Byte Order:                           Little Endian
Address sizes:                        46 bits physical, 48 bits virtual
CPU(s):                               96
On-line CPU(s) list:                  0-95
Thread(s) per core:                   2
Core(s) per socket:                   24
Socket(s):                            2
NUMA node(s):                         2
Vendor ID:                            GenuineIntel
CPU family:                           6
Model:                                85
Model name:                           Intel(R) Xeon(R) CPU @ 2.20GHz
Stepping:                             7
CPU MHz:                              2200.174
BogoMIPS:                             4400.34
Hypervisor vendor:                    KVM
Virtualization type:                  full
L1d cache:                            1.5 MiB
L1i cache:                            1.5 MiB
L2 cache:                             48 MiB
L3 cache:                             77 MiB
NUMA node0 CPU(s):                    0-23,48-71
NUMA node1 CPU(s):                    24-47,72-95
Vulnerability Gather data sampling:   Not affected
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Mitigation; Enhanced IBRS
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves arat avx512_vnni md_clear arch_capabilities

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torchaudio==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.47.0
[pip3] triton==3.1.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torch                     2.5.1                    pypi_0    pypi
[conda] torchaudio                2.5.1                    pypi_0    pypi
[conda] torchvision               0.20.1                   pypi_0    pypi
[conda] transformers              4.47.0                   pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.4.post2.dev246+g9743d64e
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
�[4mGPU0	GPU1	GPU2	GPU3	GPU4	GPU5	GPU6	GPU7	CPU Affinity	NUMA Affinity	GPU NUMA ID�[0m
GPU0	 X 	NV12	NV12	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU1	NV12	 X 	NV12	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU2	NV12	NV12	 X 	NV12	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU3	NV12	NV12	NV12	 X 	NV12	NV12	NV12	NV12	0-23,48-71	0		N/A
GPU4	NV12	NV12	NV12	NV12	 X 	NV12	NV12	NV12	24-47,72-95	1		N/A
GPU5	NV12	NV12	NV12	NV12	NV12	 X 	NV12	NV12	24-47,72-95	1		N/A
GPU6	NV12	NV12	NV12	NV12	NV12	NV12	 X 	NV12	24-47,72-95	1		N/A
GPU7	NV12	NV12	NV12	NV12	NV12	NV12	NV12	 X 	24-47,72-95	1		N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

LD_LIBRARY_PATH=/opt/conda/envs/vllm-expr/lib/python3.11/site-packages/cv2/../../lib64:/usr/local/cuda/lib64:/usr/local/nccl2/lib:/usr/local/cuda/extras/CUPTI/lib64:/usr/local/cuda/lib64:/usr/local/nccl2/lib:/usr/local/cuda/extras/CUPTI/lib64
CUDA_MODULE_LOADING=LAZY

Model Input Dumps

No response

🐛 Describe the bug

I tried to run the script vllm/benchmarks/disagg_benchmarks/disagg_performance_benchmark.sh with TP=2 for both prefill and decode instance. The prefill instance or decode instance hung after receiving several requests from the proxy server.

I received messages similar to the message below from the command line repeatedly when one of the instances hung

DEBUG 12-17 00:40:47 metrics.py:460] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
DEBUG 12-17 00:40:47 engine.py:190] Waiting for new requests in engine loop.
DEBUG 12-17 00:40:50 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:50 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:50 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:50 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:50 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:50 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:50 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:50 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:55 client.py:165] Heartbeat successful.                                                                
DEBUG 12-17 00:40:57 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:57 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:57 client.py:186] Waiting for output from MQLLMEngine.
DEBUG 12-17 00:40:57 client.py:165] Heartbeat successful.                                                                
DEBUG 12-17 00:40:57 client.py:165] Heartbeat successful.

ProcessGroupNCCL threw timeout exceptions after around 5 min:

[rank0]:[E1217 00:48:38.068038521 ProcessGroupNCCL.cpp:616] [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=10, OpType=GATHER, NumelIn=64128, NumelOut=128256, Timeout(ms)=600000) ran for 600027 milliseconds before timing
 out.                                                                                                                                                                                                                                              
[rank0]:[E1217 00:48:38.068411586 ProcessGroupNCCL.cpp:1785] [PG ID 2 PG GUID 3 Rank 0] Exception (either an error or timeout) detected by watchdog at work: 10, last enqueued NCCL work: 10, last completed NCCL work: 9.                         
[rank0]:[E1217 00:48:38.068431881 ProcessGroupNCCL.cpp:1834] [PG ID 2 PG GUID 3 Rank 0] Timeout at NCCL work: 10, last enqueued NCCL work: 10, last completed NCCL work: 9.                                                                        
[rank0]:[E1217 00:48:38.068440945 ProcessGroupNCCL.cpp:630] [Rank 0] Some NCCL operations have failed or timed out. Due to the asynchronous nature of CUDA kernels, subsequent GPU operations might run on corrupted/incomplete data.              
[rank0]:[E1217 00:48:38.068452819 ProcessGroupNCCL.cpp:636] [Rank 0] To avoid data inconsistency, we are taking the entire process down.                                                                                                           
[rank0]:[E1217 00:48:38.070968924 ProcessGroupNCCL.cpp:1595] [PG ID 2 PG GUID 3 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=10, OpType=GATHER, NumelIn=
64128, NumelOut=128256, Timeout(ms)=600000) ran for 600027 milliseconds before timing out.                                                                                                                                                         
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):                                                                                                                            
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1cd1266446 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libc10.so)                                                                                
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7f1cd2579772 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)        
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f1cd2580bb3 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)                                                                                  
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f1cd258261d in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)                                                                                 
frame #4: <unknown function> + 0x145c0 (0x7f1d1af0f5c0 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch.so)                                                                                                            
frame #5: <unknown function> + 0x7ea7 (0x7f1d1d54bea7 in /lib/x86_64-linux-gnu/libpthread.so.0)                                                                                                                                                    
frame #6: clone + 0x3f (0x7f1d1d31cacf in /lib/x86_64-linux-gnu/libc.so.6)                                                                                                                                                                         
                                                                                                                                                                                                                                                   
terminate called after throwing an instance of 'c10::DistBackendError'                                                                                                                                                                             
  what():  [PG ID 2 PG GUID 3 Rank 0] Process group watchdog thread terminated with exception: [Rank 0] Watchdog caught collective operation timeout: WorkNCCL(SeqNum=10, OpType=GATHER, NumelIn=64128, NumelOut=128256, Timeout(ms)=600000) ran fo
r 600027 milliseconds before timing out.                                                                                                                                                                                                           
Exception raised from checkTimeout at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:618 (most recent call first):                                                                                                                            
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1cd1266446 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libc10.so)                                                                                
frame #1: c10d::ProcessGroupNCCL::WorkNCCL::checkTimeout(std::optional<std::chrono::duration<long, std::ratio<1l, 1000l> > >) + 0x282 (0x7f1cd2579772 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)        
frame #2: c10d::ProcessGroupNCCL::watchdogHandler() + 0x233 (0x7f1cd2580bb3 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)                                                                                  
frame #3: c10d::ProcessGroupNCCL::ncclCommWatchdog() + 0x14d (0x7f1cd258261d in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)                                                                                 
frame #4: <unknown function> + 0x145c0 (0x7f1d1af0f5c0 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch.so)                                                                                                            
frame #5: <unknown function> + 0x7ea7 (0x7f1d1d54bea7 in /lib/x86_64-linux-gnu/libpthread.so.0)                                                                                                                                                    
frame #6: clone + 0x3f (0x7f1d1d31cacf in /lib/x86_64-linux-gnu/libc.so.6)                                                                                                                                                                         
                                                                                                                                                                                                                                                   
Exception raised from ncclCommWatchdog at ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1601 (most recent call first):                                                                                                                       
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x96 (0x7f1cd1266446 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libc10.so)                                                                                
frame #1: <unknown function> + 0xe4271b (0x7f1cd21ef71b in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch_cuda.so)                                                                                                      
frame #2: <unknown function> + 0x145c0 (0x7f1d1af0f5c0 in /opt/conda/envs/vllm-expr/lib/python3.11/site-packages/torch/lib/libtorch.so)                                                                                                            
frame #3: <unknown function> + 0x7ea7 (0x7f1d1d54bea7 in /lib/x86_64-linux-gnu/libpthread.so.0)                                                                                                                                                    
frame #4: clone + 0x3f (0x7f1d1d31cacf in /lib/x86_64-linux-gnu/libc.so.6)

Example script modified from disagg_performance_benchmark.sh:

#!/bin/bash

set -x

kill_gpu_processes() {
  # kill all processes on GPU.
  pgrep pt_main_thread | xargs -r kill -9
  pgrep python3 | xargs -r kill -9
  for port in 8000 8100 8200; do lsof -t -i:$port | xargs -r kill -9; done
  sleep 1
}

wait_for_server() {
  # wait for vllm server to start
  # return 1 if vllm server crashes
  local port=$1
  timeout 1200 bash -c "
    until curl -s localhost:${port}/v1/completions > /dev/null; do
      sleep 1
    done" && return 0 || return 1
}

launch_disagg_prefill() {
  model="meta-llama/Meta-Llama-3.1-8B-Instruct" 
  # disagg prefill
  CUDA_VISIBLE_DEVICES=0,1 python3 \
    -m vllm.entrypoints.openai.api_server \
    --model $model \
    -tp 2 \
    --port 8100 \
    --max-model-len 10000 \
    --disable_log_requests \
    --gpu-memory-utilization 0.6 \
    --kv-transfer-config \
    '{"kv_connector":"PyNcclConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":1e9}' &

  CUDA_VISIBLE_DEVICES=2,3 python3 \
    -m vllm.entrypoints.openai.api_server \
    --model $model \
    -tp 2 \
    --port 8200 \
    --max-model-len 10000 \
    --disable_log_requests \
    --gpu-memory-utilization 0.6 \
    --kv-transfer-config \
    '{"kv_connector":"PyNcclConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":1e9}' &

  wait_for_server 8100
  wait_for_server 8200
  python3 disagg_prefill_proxy_server.py &
  sleep 1
}

benchmark() {
  results_folder="./results"
  model="meta-llama/Meta-Llama-3.1-8B-Instruct"
  dataset_name="sonnet"
  dataset_path="../sonnet_4x.txt"
  num_prompts=100
  qps=$1
  prefix_len=50
  input_len=1024
  output_len=$2
  tag=$3

  python3 ../benchmark_serving.py \
          --backend vllm \
          --model $model \
          --dataset-name $dataset_name \
          --dataset-path $dataset_path \
          --sonnet-input-len $input_len \
          --sonnet-output-len "$output_len" \
          --sonnet-prefix-len $prefix_len \
          --num-prompts $num_prompts \
          --port 8000 \
          --save-result \
          --result-dir $results_folder \
          --result-filename "$tag"-qps-"$qps".json \
          --request-rate "$qps"

  sleep 2

}

main() {

  (which wget && which curl) || (apt-get update && apt-get install -y wget curl)
  (which jq) || (apt-get -y install jq)
  (which socat) || (apt-get -y install socat)

  pip install quart httpx matplotlib aiohttp

  cd "$(dirname "$0")"

  export VLLM_LOGGING_LEVEL=DEBUG
  export VLLM_TRACE_FUNCTION=1

  cd ..
  # create sonnet-4x.txt so that we can sample 2048 tokens for input
  echo "" > sonnet_4x.txt
  for _ in {1..4}
  do
    cat sonnet.txt >> sonnet_4x.txt
  done
  cd disagg_benchmarks

  rm -rf results
  mkdir results

  default_output_len=6

  export VLLM_HOST_IP=$(hostname -I | awk '{print $1}')

  qps=8

  launch_disagg_prefill
  benchmark $qps $default_output_len disagg_prefill
  kill_gpu_processes

}

main "$@"

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@Louis-99 Louis-99 added the bug Something isn't working label Dec 17, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@Louis-99