add support for 4bit inference

[iMountTai]

Description

This PR adds support for 4-bit inference, which can effectively reduce the VRAM. The VRAM after loading the model is as follows (in Tesla P40) :

load_in_4bit	load_in_8bit	fp16
4999M	7989M	13759M

Usage
Just add the --load_in_4bit to the launching command. For example:

python scripts/inference/inference_hf.py \
    --base_model path_to_merged_llama2_or_alpaca2_hf_dir \
    --with_prompt \
    --load_in_4bit \
    --interactive

Related Issue

None.

[airaria]

1. Update requirements.txt
2. Since this is a breaking change, please make sure all the occurrences of load_in_8bit in Wiki and scripts are replaced with load_in_kbit 8
3. Can load_in_kbit be added to openai_api_server.py and openai_api_server_vllm.py?

[iMountTai]

@airaria

• The requirements and wiki have been updated, and the load_in_8bit setting has been retained.
• load_in_kbit has been added to openai_api_server.py. openai_api_server_vllm.py does not surpport load_in_kbit

[airaria]

TODO: Update wiki w.r.t. this PR. @iMountTai