[Bug]: Function calling with Qwen & Streaming ('NoneType' object has no attribute 'get')

[githebs]

Your current environment

The output of `python collect_env.py`

Your output of `python collect_env.py` here

Model Input Dumps

No response

🐛 Describe the bug

vLLM Version

v0.6.3.post1

Model

Qwen2.5-7B-Instruct

Docker command for vLLM

command: --host 0.0.0.0 --model /hf/Qwen-Qwen2.5-7B-Instruct --max-model-len 32768 --gpu_memory_utilization 0.9 --enable-auto-tool-choice --tool-call-parser hermes

Parsing from my own fastapi

async def stream_response(payload: dict, log: RequestLogger) -> AsyncGenerator[str, None]:
    """Handle streaming response from vLLM."""
    async with httpx.AsyncClient() as client:
        try:
            async with client.stream(
                'POST',
                VLLM_API_BASE,
                json=payload,
                headers={"Content-Type": "application/json"},
                timeout=30.0
            ) as response:
                if response.status_code != 200:
                    error_msg = f"vLLM API error: {response.status_code}"
                    log(error_msg, level='error')
                    yield f"data: {json.dumps({'error': error_msg})}\n\n"
                    return

                async for line in response.aiter_lines():
                    if not line or not line.startswith('data: '):
                        continue
                        
                    line = line.removeprefix('data: ')
                    if line.strip() == '[DONE]':
                        log("Stream completed")
                        yield 'data: [DONE]\n\n'
                        break
                    
                    try:
                        parsed = json.loads(line)
                        log("Streaming chunk", parsed)

                        # Handle tool calls in streaming response
                        if 'choices' in parsed and parsed['choices']:
                            choice = parsed['choices'][0]
                            if 'delta' in choice and 'tool_calls' in choice['delta']:
                                tool_call = choice['delta']['tool_calls'][0]
                                
                                if ('function' in tool_call and 
                                    'name' in tool_call['function'] and 
                                    'arguments' in tool_call['function']):
                                    
                                    func_name = tool_call['function']['name']
                                    args = json.loads(tool_call['function']['arguments'])
                                    
                                    if func_name == 'add_numbers':
                                        result = add_numbers(args['a'], args['b'])
                                        yield f'data: {json.dumps({"choices": [{"delta": {"content": str(result)}}]})}\n\n'
                                        continue

                        yield f'data: {line}\n\n'
                    except json.JSONDecodeError as e:
                        log(f"Failed to parse streaming response: {str(e)}", level='error')
                        continue

        except httpx.RequestError as e:
            error_msg = f"Streaming request failed: {str(e)}"
            log(error_msg, level='error')
            yield f"data: {json.dumps({'error': error_msg})}\n\n"
        
    log("Stream connection closed")

vLLM error

vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] Error trying to handle streaming tool call. vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] Traceback (most recent call last): vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py", line 226, in extract_tool_calls_streaming vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] function_name: Union[str, None] = current_tool_call.get("name") vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] ^^^^^^^^^^^^^^^^^^^^^ vllm | ERROR 10-30 14:55:01 hermes_tool_parser.py:337] AttributeError: 'NoneType' object has no attribute 'get'

Please note that everything works if

1. Streaming with no tools
2. Not streaming with tools

Any guidance ?
Thanks in advance everyone

PS: I have seen the posts from #9693 but my issue seems different since i actually use a "supported" model.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

cc @K-Mistele

[K-Mistele]

Thanks for the ping @DarkLight1337
@githebs can you share a request configuration that reproduces the issue consistently (temperature=0 is great for reproducibility, but no worries if you need a higher temp and it only happens sometimes) so that I can debug and take a look?

[K-Mistele]

Hi @githebs - we have had a discussion on this issue in #9693. Please see my comment here and let me know if this seems like a good path forward for you.

[K-Mistele]

Please check #9908 :)

[frei-x]

Stream output, if the function has no parameters, an error will be reported directly

[K-Mistele]

Stream output, if the function has no parameters, an error will be reported directly

Yeah, this is what I'm thinking too. #9908 (comment)

[githebs]

@frei-x @K-Mistele

thanks for the answer, sorry for the delay, I answered in the PR here #9908 (comment) but basically, yes, if the argument is blank, it doesn't work