TypeError: llama_causal_lm_forward() got an unexpected keyword argument 'shard_config'

[willy808]

i run example/inference code to check to throughput in our machine but we got above errors
Package Version Editable project location

---

accelerate 0.30.1
aiosignal 1.3.1
annotated-types 0.6.0
apex 0.1
attrs 23.2.0
bcrypt 4.0.1
beautifulsoup4 4.12.3
bitsandbytes 0.43.1
boltons 23.0.0
brotlipy 0.7.0
certifi 2023.5.7
cffi 1.15.0
cfgv 3.3.1
charset-normalizer 2.0.4
click 8.1.3
cmake 3.29.3
colorama 0.4.4
colossalai 0.3.7 /home/phison/ColossalAI
colossalai-nightly 2024.5.11
conda 23.3.1
conda-content-trust 0+unknown
conda-package-handling 1.8.1
contexttimer 0.3.3
cryptography 36.0.0
distlib 0.3.6
einops 0.8.0
fabric 3.0.1
filelock 3.12.0
flash-attn 2.5.8
frozenlist 1.4.1
fsspec 2024.3.1
galore-torch 1.0
google 3.0.0
huggingface-hub 0.23.0
identify 2.5.24
idna 3.3
invoke 2.1.2
Jinja2 3.1.4
jsonpatch 1.32
jsonpointer 2.1
jsonschema 4.22.0
jsonschema-specifications 2023.12.1
lightllm 2.0.0 /home/phison/ColossalAI/apex/lightllm
lit 18.1.4
markdown-it-py 2.2.0
MarkupSafe 2.1.5
mdurl 0.1.2
mkl-fft 1.3.1
mkl-random 1.2.2
mkl-service 2.4.0
mpmath 1.3.0
msgpack 1.0.8
networkx 3.2.1
ninja 1.11.1
nodeenv 1.8.0
numpy 1.22.3
nvidia-cublas-cu11 11.10.3.66
nvidia-cublas-cu12 12.1.3.1
nvidia-cuda-cupti-cu11 11.7.101
nvidia-cuda-cupti-cu12 12.1.105
nvidia-cuda-nvrtc-cu11 11.7.99
nvidia-cuda-nvrtc-cu12 12.1.105
nvidia-cuda-runtime-cu11 11.7.99
nvidia-cuda-runtime-cu12 12.1.105
nvidia-cudnn-cu11 8.5.0.96
nvidia-cudnn-cu12 8.9.2.26
nvidia-cufft-cu11 10.9.0.58
nvidia-cufft-cu12 11.0.2.54
nvidia-curand-cu11 10.2.10.91
nvidia-curand-cu12 10.3.2.106
nvidia-cusolver-cu11 11.4.0.1
nvidia-cusolver-cu12 11.4.5.107
nvidia-cusparse-cu11 11.7.4.91
nvidia-cusparse-cu12 12.1.0.106
nvidia-nccl-cu11 2.14.3
nvidia-nccl-cu12 2.18.1
nvidia-nvjitlink-cu12 12.4.127
nvidia-nvtx-cu11 11.7.91
nvidia-nvtx-cu12 12.1.105
packaging 23.0
paramiko 3.1.0
peft 0.10.0
Pillow 9.0.1
pip 24.0
platformdirs 3.5.1
pluggy 1.0.0
plumbum 1.8.3
pre-commit 3.3.2
protobuf 5.26.1
psutil 5.9.5
pycosat 0.6.3
pycparser 2.21
pydantic 2.7.1
pydantic_core 2.18.2
Pygments 2.15.1
PyNaCl 1.5.0
pyOpenSSL 22.0.0
PySocks 1.7.1
PyYAML 6.0
pyzmq 26.0.3
ray 2.22.0
referencing 0.35.1
regex 2024.5.10
requests 2.27.1
rich 13.3.5
rpds-py 0.18.1
rpyc 6.0.0
ruamel.yaml 0.16.12
ruamel.yaml.clib 0.2.6
ruamel-yaml-conda 0.15.100
safetensors 0.4.3
sentencepiece 0.2.0
setuptools 69.5.1
six 1.16.0
soupsieve 2.5
sympy 1.12
tensornvme 0.1.0
titans 0.0.7
tokenizers 0.15.2
toolz 0.12.0
torch 2.1.1
torchaudio 2.1.1
torchvision 0.16.1
tqdm 4.63.0
transformers 4.36.2
triton 2.1.0
triton-nightly 3.0.0.post20240424212437
typing_extensions 4.11.0
urllib3 1.26.8
uvloop 0.19.0
virtualenv 20.23.0
wheel 0.43.0
please install Colossal-AI from https://www.colossalai.org/download or from source
/opt/conda/lib/python3.9/site-packages/colossalai/shardformer/layer/normalization.py:45: UserWarning: Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel
warnings.warn("Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel")
please install lightllm from source to run inference: https://github.com/ModelTC/lightllm
please install lightllm from source to run inference: https://github.com/ModelTC/lightllm
you should install triton from https://github.com/openai/triton
please install Colossal-AI from https://www.colossalai.org/download or from source
/opt/conda/lib/python3.9/site-packages/colossalai/shardformer/layer/normalization.py:45: UserWarning: Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel
warnings.warn("Please install apex from source (https://github.com/NVIDIA/apex) to use the fused layernorm kernel")
please install lightllm from source to run inference: https://github.com/ModelTC/lightllm
please install lightllm from source to run inference: https://github.com/ModelTC/lightllm
you should install triton from https://github.com/openai/triton
[05/15/24 07:15:31] INFO colossalai - colossalai - INFO: /opt/conda/lib/python3.9/site-packages/colossalai/initialize.py:60 launch
INFO colossalai - colossalai - INFO: Distributed environment is initialized, world size: 1
Loading checkpoint shards: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 14/14 [00:07<00:00, 1.80it/s]/opt/conda/lib/python3.9/site-packages/transformers/generation/configuration_utils.py:389: UserWarning: do_sample is set to False. However, temperature is set to 0.3 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset temperature. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
warnings.warn(
/opt/conda/lib/python3.9/site-packages/transformers/generation/configuration_utils.py:394: UserWarning: do_sample is set to False. However, top_p is set to 0.95 -- this flag is only used in sample-based generation modes. You should set do_sample=True or unset top_p. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.
warnings.warn(
/opt/conda/lib/python3.9/site-packages/colossalai/shardformer/policies/llama.py:75: UserWarning: For llama, sequence parallelism is currently not compatible with pipeline parallelism, set to be False
warnings.warn(
Traceback (most recent call last):
File "/home/phison/ColossalAI/examples/inference/run_llama_inference.py", line 98, in
spawn(run_tp_pipeline_inference, nprocs=args.tp_size * args.pp_size, args=args)
File "/opt/conda/lib/python3.9/site-packages/colossalai/testing/utils.py", line 252, in spawn
mp.spawn(wrapped_func, nprocs=nprocs)
File "/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 246, in spawn
return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
File "/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 202, in start_processes
while not context.join():
File "/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 163, in join
raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException:

-- Process 0 terminated with the following error:
Traceback (most recent call last):
File "/opt/conda/lib/python3.9/site-packages/torch/multiprocessing/spawn.py", line 74, in _wrap
fn(i, *args)
File "/home/phison/ColossalAI/examples/inference/run_llama_inference.py", line 72, in run_tp_pipeline_inference
run_inference(args)
File "/home/phison/ColossalAI/examples/inference/run_llama_inference.py", line 61, in run_inference
outputs = engine.generate(inputs)
File "/opt/conda/lib/python3.9/site-packages/colossalai/inference/engine/engine.py", line 142, in generate
out, timestamp = self.schedule.generate_step(self.model, iter([input_list]))
File "/opt/conda/lib/python3.9/site-packages/colossalai/pipeline/schedule/generate.py", line 260, in generate_step
return self.generate_step_one_stage(model, data_iter)
File "/opt/conda/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/colossalai/pipeline/schedule/generate.py", line 292, in generate_step_one_stage
action()
File "/opt/conda/lib/python3.9/site-packages/colossalai/pipeline/schedule/generate.py", line 149, in _load_stage_action
output_dict = model_forward(model, inputs_dict, interval_inputs)
File "/opt/conda/lib/python3.9/site-packages/colossalai/pipeline/schedule/_utils.py", line 120, in model_forward
return model(**data, **internal_inputs)
File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
TypeError: llama_causal_lm_forward() got an unexpected keyword argument 'shard_config'