[Usage]: How to set a dynamic temperature when sampling? (decrease gradually as more tokens generated)

[StarDewXXX]

How would you like to use vllm

I want to set a dynamic temperature when sampling. Specifically, I need to dynamically adjust the temperature coefficient based on the length of the currently generated token each time the logits are processed. It seems that sampling_params cannot achieve this feature. Which part of the code should I modify?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

Get the temperature from sampling_params

here:

vllm/vllm/model_executor/sampling_metadata.py

Line 403 in f04e407

temperature = sampling_params.temperature

but I guess vllm has a caching mechanism.

Maybe it needs to be closed or I don’t know any good way bypass.

https://github.com/vllm-project/vllm/blob/f04e407e6b6b9ce65c16cffda836f05c2ad32682/vllm/model_executor/layers/sampler.py#L241C1-L250C1

Modifying such deep code is always hard, good luck