[Bug]: Hermes tool parser output error stream arguments in some cases.

[xiyuan-lee]

Your current environment

The output of `python collect_env.py`

WARNING 11-16 23:18:45 _custom_ops.py:20] Failed to import from vllm._C with ModuleNotFoundError("No module named 'vllm._C'")
/home/oem/repo/vllm/vllm/connections.py:8: RuntimeWarning: Failed to read commit hash:
No module named 'vllm._version'
  from vllm.version import __version__ as VLLM_VERSION
Collecting environment information...
/home/oem/anaconda3/envs/vllm-test/lib/python3.10/site-packages/torch/cuda/__init__.py:129: UserWarning: CUDA initialization: The NVIDIA driver on your system is too old (found version 11040). Please update your GPU driver by downloading and installing a new version from the URL: http://www.nvidia.com/Download/index.aspx Alternatively, go to: https://pytorch.org to install a PyTorch version that has been compiled with your version of the CUDA driver. (Triggered internally at ../c10/cuda/CUDAFunctions.cpp:108.)
  return torch._C._cuda_getDeviceCount() > 0
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 20.04.6 LTS (x86_64)
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.31

Python version: 3.10.15 (main, Oct  3 2024, 07:27:34) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-glibc2.31
Is CUDA available: False
CUDA runtime version: No CUDA
CUDA_MODULE_LOADING set to: N/A
GPU models and configuration: No CUDA
Nvidia driver version: No CUDA
cuDNN version: No CUDA
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   46 bits physical, 48 bits virtual
CPU(s):                          20
On-line CPU(s) list:             0-19
Thread(s) per core:              2
Core(s) per socket:              10
Socket(s):                       1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           154
Model name:                      12th Gen Intel(R) Core(TM) i9-12900H
Stepping:                        3
CPU MHz:                         2918.408
BogoMIPS:                        5836.81
Virtualization:                  VT-x
Hypervisor vendor:               Microsoft
Virtualization type:             full
L1d cache:                       480 KiB
L1i cache:                       320 KiB
L2 cache:                        12.5 MiB
L3 cache:                        24 MiB
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm md_clear serialize flush_l1d arch_capabilities

Versions of relevant libraries:
[pip3] mypy==1.11.1
[pip3] mypy-extensions==1.0.0
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] sentence-transformers==3.2.1
[pip3] torch==2.5.1
[pip3] torchvision==0.20.1
[pip3] transformers==4.45.2
[pip3] transformers-stream-generator==0.0.5
[pip3] triton==3.1.0
[conda] numpy                     1.26.4                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] sentence-transformers     3.2.1                    pypi_0    pypi
[conda] torch                     2.5.1                    pypi_0    pypi
[conda] torchvision               0.20.1                   pypi_0    pypi
[conda] transformers              4.45.2                   pypi_0    pypi
[conda] transformers-stream-generator 0.0.5                    pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A (dev)
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
Could not collect

LD_LIBRARY_PATH=/home/oem/anaconda3/envs/vllm-test/lib/python3.10/site-packages/cv2/../../lib64:

🐛 Describe the bug

The extract_tool_calls_streaming component of the Hermes tool parser generates error arguments during the parsing of streaming tool function outputs when the LLM produces a specific output content.

LLM

Qwen2.5-72B-Instruct-AWQ

The docker-compose file to run the LLM:

version: '3.8'

services:
  Qwen72BAWQ:
    image: vllm/vllm-openai:v0.6.3.post1
    entrypoint: ["python3","-u", "-m","vllm.entrypoints.openai.api_server", "--served-model-name","Qwen2.5-72B-Instruct-AWQ","--model","/data/models/Qwen2.5-72B-Instruct-AWQ", "--enable-auto-tool-choice", "--tool-call-parser", "hermes", "--enforce-eager",  "--tensor-parallel-size", "1",   "--gpu_memory_utilization", "0.97" ]

    environment:
      - "CUDA_VISIBLE_DEVICES=1"
    restart: always
    ports:
      - "8000:8000"
    volumes:
      - /data/models:/data/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

vLLM server log

  | INFO 11-16 08:15:10 logger.py:37] Received request chat-a183cc0045ba4201833c24cd2be5c43e: prompt: '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city to find the weather for, e.g. \'San Francisco\'"}, "state": {"type": "string", "description": "the two-letter abbreviation for the state that the city is in, e.g. \'CA\' which would mean \'California\'"}, "unit": {"type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"]}}, "required": ["city", "state", "unit"]}}}\n{"type": "function", "function": {"name": "predict_x", "description": "predict x using wd, sd, injection and module_id.", "parameters": {"type": "object", "properties": {"wd": {"type": "number", "description": "shi du"}, "sd": {"type": "number", "description": "wen du"}, "injection": {"type": "string", "description": "injection type"}, "module_id": {"type": "string", "description": "module id"}}, "required": ["wd", "sd", "injection", "module_id"]}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": <function-name>, "arguments": <args-json-object>}\n</tool_call><|im_end|>\n<|im_start|>user\nHi! How are you doing today?<|im_end|>\n<|im_start|>assistant\nI\'m doing well! How can I help you?<|im_end|>\n<|im_start|>user\nwhen wd=23, sd=0.8, injection=ZSJB39, module_id=M62121G9142, what\'s x ? <|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32306, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), guided_decoding=GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None), prompt_token_ids: [......], lora_request: None, prompt_adapter_request: None.
  | DEBUG 11-16 08:15:10 async_llm_engine.py:523] Building guided decoding logits processor. Params: GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None)
  | DEBUG 11-16 08:15:11 client.py:154] Heartbeat successful.
  | INFO 11-16 08:15:11 engine.py:290] Added request chat-a183cc0045ba4201833c24cd2be5c43e.
  | INFO 11-16 08:15:11 metrics.py:349] Avg prompt throughput: 65.4 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.5%, CPU KV cache usage: 0.0%.
  | DEBUG 11-16 08:15:12 client.py:154] Heartbeat successful.
  | DEBUG 11-16 08:15:13 llm_engine.py:1460] Stopping remote worker execution loop.
  | INFO:     10.182.93.184:54706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
  | INFO 11-16 08:15:14 logger.py:37] Received request chat-5fca4945ed9844eabbe21db8e2de215d: prompt: '<|im_start|>system\nYou are Qwen, created by Alibaba Cloud. You are a helpful assistant.\n\n# Tools\n\nYou may call one or more functions to assist with the user query.\n\nYou are provided with function signatures within <tools></tools> XML tags:\n<tools>\n{"type": "function", "function": {"name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": {"type": "object", "properties": {"city": {"type": "string", "description": "The city to find the weather for, e.g. \'San Francisco\'"}, "state": {"type": "string", "description": "the two-letter abbreviation for the state that the city is in, e.g. \'CA\' which would mean \'California\'"}, "unit": {"type": "string", "description": "The unit to fetch the temperature in", "enum": ["celsius", "fahrenheit"]}}, "required": ["city", "state", "unit"]}}}\n{"type": "function", "function": {"name": "predict_x", "description": "predict x using wd, sd, injection and module_id.", "parameters": {"type": "object", "properties": {"wd": {"type": "number", "description": "shi du"}, "sd": {"type": "number", "description": "wen du"}, "injection": {"type": "string", "description": "injection type"}, "module_id": {"type": "string", "description": "module id"}}, "required": ["wd", "sd", "injection", "module_id"]}}}\n</tools>\n\nFor each function call, return a json object with function name and arguments within <tool_call></tool_call> XML tags:\n<tool_call>\n{"name": <function-name>, "arguments": <args-json-object>}\n</tool_call><|im_end|>\n<|im_start|>user\nHi! How are you doing today?<|im_end|>\n<|im_start|>assistant\nI\'m doing well! How can I help you?<|im_end|>\n<|im_start|>user\nwhen wd=23, sd=0.8, injection=ZSJB39, module_id=M62121G9142, what\'s x ? <|im_end|>\n<|im_start|>assistant\n', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=0.7, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=32306, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None), guided_decoding=GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None), prompt_token_ids: [......], lora_request: None, prompt_adapter_request: None.
  | INFO:     10.182.93.184:54706 - "POST /v1/chat/completions HTTP/1.1" 200 OK
async_llm_engine.py:523] Building guided decoding logits processor. Params: GuidedDecodingParams(json=None, regex=None, choice=None, grammar=None, json_object=None, backend=None, whitespace_pattern=None)
  | INFO 11-16 08:15:14 engine.py:290] Added request chat-5fca4945ed9844eabbe21db8e2de215d.
hermes_tool_parser.py:124] delta_text: <tool_call>
hermes_tool_parser.py:125] delta_token_ids: [151657]
hermes_tool_parser.py:175] Starting on a new tool 0
hermes_tool_parser.py:218] Parsed tool call None
  | ERROR 11-16 08:15:14 hermes_tool_parser.py:337] Error trying to handle streaming tool call.
  | ERROR 11-16 08:15:14 hermes_tool_parser.py:337] Traceback (most recent call last):
  | ERROR 11-16 08:15:14 hermes_tool_parser.py:337]   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/tool_parsers/hermes_tool_parser.py", line 226, in extract_tool_calls_streaming
  | ERROR 11-16 08:15:14 hermes_tool_parser.py:337]     function_name: Union[str, None] = current_tool_call.get("name")
  | ERROR 11-16 08:15:14 hermes_tool_parser.py:337]                                       ^^^^^^^^^^^^^^^^^^^^^
  | ERROR 11-16 08:15:14 hermes_tool_parser.py:337] AttributeError: 'NoneType' object has no attribute 'get'
hermes_tool_parser.py:124] delta_text:
hermes_tool_parser.py:124]
hermes_tool_parser.py:125] delta_token_ids: [198]
hermes_tool_parser.py:220] not enough tokens to parse into JSON yet
hermes_tool_parser.py:124] delta_text: {"
hermes_tool_parser.py:125] delta_token_ids: [4913]
hermes_tool_parser.py:218] Parsed tool call {}
hermes_tool_parser.py:124] delta_text: name
hermes_tool_parser.py:125] delta_token_ids: [606]
hermes_tool_parser.py:218] Parsed tool call {}
hermes_tool_parser.py:124] delta_text: ":
hermes_tool_parser.py:125] delta_token_ids: [788]
hermes_tool_parser.py:218] Parsed tool call {}
hermes_tool_parser.py:124] delta_text:  "
hermes_tool_parser.py:125] delta_token_ids: [330]
hermes_tool_parser.py:218] Parsed tool call {}
hermes_tool_parser.py:124] delta_text: predict
hermes_tool_parser.py:125] delta_token_ids: [34698]
hermes_tool_parser.py:218] Parsed tool call {}
hermes_tool_parser.py:124] delta_text: _x
hermes_tool_parser.py:125] delta_token_ids: [3212]
hermes_tool_parser.py:218] Parsed tool call {}
hermes_tool_parser.py:124] delta_text: ",
hermes_tool_parser.py:125] delta_token_ids: [497]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x'}
hermes_tool_parser.py:124] delta_text:  "
hermes_tool_parser.py:125] delta_token_ids: [330]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x'}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: None
hermes_tool_parser.py:267] against new ones: None
hermes_tool_parser.py:271] Skipping text  " - no arguments
hermes_tool_parser.py:124] delta_text: arguments
hermes_tool_parser.py:125] delta_token_ids: [16370]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x'}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: None
hermes_tool_parser.py:267] against new ones: None
hermes_tool_parser.py:271] Skipping text arguments - no arguments
client.py:154] Heartbeat successful.
hermes_tool_parser.py:124] delta_text: ":
hermes_tool_parser.py:125] delta_token_ids: [788]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x'}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: None
hermes_tool_parser.py:267] against new ones: None
hermes_tool_parser.py:271] Skipping text ": - no arguments
hermes_tool_parser.py:124] delta_text:  {"
hermes_tool_parser.py:125] delta_token_ids: [5212]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: None
hermes_tool_parser.py:267] against new ones: {}
hermes_tool_parser.py:271] Skipping text  {" - no arguments
hermes_tool_parser.py:124] delta_text: wd
hermes_tool_parser.py:125] delta_token_ids: [6377]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {}
hermes_tool_parser.py:267] against new ones: {}
hermes_tool_parser.py:271] Skipping text wd - no arguments
hermes_tool_parser.py:124] delta_text: ":
hermes_tool_parser.py:125] delta_token_ids: [788]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {}
hermes_tool_parser.py:267] against new ones: {}
hermes_tool_parser.py:271] Skipping text ": - no arguments
hermes_tool_parser.py:124] delta_text:
hermes_tool_parser.py:125] delta_token_ids: [220]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {}
hermes_tool_parser.py:267] against new ones: {}
hermes_tool_parser.py:271] Skipping text   - no arguments
hermes_tool_parser.py:124] delta_text: 2
hermes_tool_parser.py:125] delta_token_ids: [17]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 2}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {}
hermes_tool_parser.py:267] against new ones: {'wd': 2}
hermes_tool_parser.py:286] finding 2 in {"wd": 2}
hermes_tool_parser.py:295] First tokens in arguments received: {"wd": 2
hermes_tool_parser.py:124] delta_text: 3
hermes_tool_parser.py:125] delta_token_ids: [18]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 2}
hermes_tool_parser.py:267] against new ones: {'wd': 23}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 2}
hermes_tool_parser.py:316] got argument diff 3
hermes_tool_parser.py:124] delta_text: ,
hermes_tool_parser.py:125] delta_token_ids: [11]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23}
hermes_tool_parser.py:267] against new ones: {'wd': 23}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text:  "
hermes_tool_parser.py:125] delta_token_ids: [330]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23}
hermes_tool_parser.py:267] against new ones: {'wd': 23}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: sd
hermes_tool_parser.py:125] delta_token_ids: [13446]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23}
hermes_tool_parser.py:267] against new ones: {'wd': 23}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: ":
hermes_tool_parser.py:125] delta_token_ids: [788]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23}
hermes_tool_parser.py:267] against new ones: {'wd': 23}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text:
hermes_tool_parser.py:125] delta_token_ids: [220]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23}
hermes_tool_parser.py:267] against new ones: {'wd': 23}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: 0
hermes_tool_parser.py:125] delta_token_ids: [15]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23}
hermes_tool_parser.py:316] got argument diff , "sd": 0
hermes_tool_parser.py:124] delta_text: .
hermes_tool_parser.py:125] delta_token_ids: [13]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: 8
hermes_tool_parser.py:125] delta_token_ids: [23]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0}
hermes_tool_parser.py:316] got argument diff .8
hermes_tool_parser.py:124] delta_text: ,
hermes_tool_parser.py:125] delta_token_ids: [11]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text:  "
hermes_tool_parser.py:125] delta_token_ids: [330]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: in
hermes_tool_parser.py:125] delta_token_ids: [258]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: jection
hermes_tool_parser.py:125] delta_token_ids: [7606]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: ":
hermes_tool_parser.py:125] delta_token_ids: [788]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8}
hermes_tool_parser.py:316] got argument diff

hermes_tool_parser.py:124] delta_text:  "
                                       🐛

                                                                                              
hermes_tool_parser.py:125] delta_token_ids: [330]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': ''}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': ''}
hermes_tool_parser.py:312] Searching for diff between


hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": ""}


hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8}


hermes_tool_parser.py:316] got argument diff , "injection": ""
                                                             🐛


hermes_tool_parser.py:124] delta_text: Z
hermes_tool_parser.py:125] delta_token_ids: [57]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'Z'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': ''}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'Z'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "Z"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": ""}
hermes_tool_parser.py:316] got argument diff Z
hermes_tool_parser.py:124] delta_text: S
hermes_tool_parser.py:125] delta_token_ids: [50]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZS'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'Z'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZS'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZS"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "Z"}
hermes_tool_parser.py:316] got argument diff S
hermes_tool_parser.py:124] delta_text: JB
hermes_tool_parser.py:125] delta_token_ids: [46107]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZS'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZS"}
hermes_tool_parser.py:316] got argument diff JB
hermes_tool_parser.py:124] delta_text: 3
hermes_tool_parser.py:125] delta_token_ids: [18]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB3'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB3'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB3"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB"}
hermes_tool_parser.py:316] got argument diff 3
hermes_tool_parser.py:124] delta_text: 9
hermes_tool_parser.py:125] delta_token_ids: [24]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB3'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB3"}
hermes_tool_parser.py:316] got argument diff 9
hermes_tool_parser.py:124] delta_text: ",
hermes_tool_parser.py:125] delta_token_ids: [497]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text:  "
hermes_tool_parser.py:125] delta_token_ids: [330]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: module
hermes_tool_parser.py:125] delta_token_ids: [4352]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: _id
hermes_tool_parser.py:125] delta_token_ids: [842]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: ":
hermes_tool_parser.py:125] delta_token_ids: [788]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text:  "
hermes_tool_parser.py:125] delta_token_ids: [330]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': ''}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': ''}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": ""}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39"}
hermes_tool_parser.py:316] got argument diff ", "module_id": "
hermes_tool_parser.py:124] delta_text: M
hermes_tool_parser.py:125] delta_token_ids: [44]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': ''}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": ""}
hermes_tool_parser.py:316] got argument diff M
hermes_tool_parser.py:124] delta_text: 6
hermes_tool_parser.py:125] delta_token_ids: [21]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M6'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M6'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M6"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M"}
hermes_tool_parser.py:316] got argument diff 6
hermes_tool_parser.py:124] delta_text: 2
hermes_tool_parser.py:125] delta_token_ids: [17]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M6'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M6"}
hermes_tool_parser.py:316] got argument diff 2
hermes_tool_parser.py:124] delta_text: 1
hermes_tool_parser.py:125] delta_token_ids: [16]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M621'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M621'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M621"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62"}
hermes_tool_parser.py:316] got argument diff 1
hermes_tool_parser.py:124] delta_text: 2
hermes_tool_parser.py:125] delta_token_ids: [17]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M6212'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M621'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M6212'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M6212"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M621"}
hermes_tool_parser.py:316] got argument diff 2
hermes_tool_parser.py:124] delta_text: 1
hermes_tool_parser.py:125] delta_token_ids: [16]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M6212'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M6212"}
hermes_tool_parser.py:316] got argument diff 1
hermes_tool_parser.py:124] delta_text: G
hermes_tool_parser.py:125] delta_token_ids: [38]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121"}
hermes_tool_parser.py:316] got argument diff G
hermes_tool_parser.py:124] delta_text: 9
hermes_tool_parser.py:125] delta_token_ids: [24]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G9'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G9'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G9"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G"}
hermes_tool_parser.py:316] got argument diff 9
hermes_tool_parser.py:124] delta_text: 1
hermes_tool_parser.py:125] delta_token_ids: [16]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G91'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G9'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G91'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G91"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G9"}
hermes_tool_parser.py:316] got argument diff 1
hermes_tool_parser.py:124] delta_text: 4
hermes_tool_parser.py:125] delta_token_ids: [19]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G914'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G91'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G914'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G914"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G91"}
hermes_tool_parser.py:316] got argument diff 4
hermes_tool_parser.py:124] delta_text: 2
hermes_tool_parser.py:125] delta_token_ids: [17]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G9142'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G914'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G9142'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G9142"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G914"}
hermes_tool_parser.py:316] got argument diff 2
hermes_tool_parser.py:124] delta_text: "}}
hermes_tool_parser.py:124]
hermes_tool_parser.py:125] delta_token_ids: [95642]
hermes_tool_parser.py:218] Parsed tool call {'name': 'predict_x', 'arguments': {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G9142'}}
hermes_tool_parser.py:252] Trying to parse current tool call with ID 0
hermes_tool_parser.py:266] diffing old arguments: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G9142'}
hermes_tool_parser.py:267] against new ones: {'wd': 23, 'sd': 0.8, 'injection': 'ZSJB39', 'module_id': 'M62121G9142'}
hermes_tool_parser.py:312] Searching for diff between
hermes_tool_parser.py:312] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G9142"}
hermes_tool_parser.py:313] and
hermes_tool_parser.py:313] {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G9142"}
hermes_tool_parser.py:316] got argument diff
hermes_tool_parser.py:124] delta_text: </tool_call>
hermes_tool_parser.py:125] delta_token_ids: [151658]


hermes_tool_parser.py:194] Finishing tool and found diff that had not been streamed yet: {"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G9142"}


llm_engine.py:1460] Stopping remote worker execution loop.
hermes_tool_parser.py:124] delta_text:
hermes_tool_parser.py:125] delta_token_ids: [151645]
hermes_tool_parser.py:147] Generating text content! skipping tool parsing.
  | DEBUG 11-16 08:15:26 client.py:170] Waiting for output from MQLLMEngine.
  | INFO 11-16 08:15:26 metrics.py:349] Avg prompt throughput: 32.0 tokens/s, Avg generation throughput: 7.8 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
  | DEBUG 11-16 08:15:26 engine.py:213] Waiting for new requests in engine loop.
  | DEBUG 11-16 08:15:26 client.py:154] Heartbeat successful.

Client code

import json

from openai import OpenAI

# Modify OpenAI's API key and API base to use vLLM's API server.

openai_api_key = "EMPTY"
openai_api_base = "http://localhost:8000/v1"

client = OpenAI(
    # defaults to os.environ.get("OPENAI_API_KEY")
    api_key=openai_api_key,
    base_url=openai_api_base,
)

models = client.models.list()
model = models.data[0].id

print(f"{model=}")

tools = [{
    "type": "function",
    "function": {
        "name": "get_current_weather",
        "description": "Get the current weather in a given location",
        "parameters": {
            "type": "object",
            "properties": {
                "city": {
                    "type":
                    "string",
                    "description":
                    "The city to find the weather for, e.g. 'San Francisco'"
                },
                "state": {
                    "type":
                    "string",
                    "description":
                    "the two-letter abbreviation for the state that the city is"
                    " in, e.g. 'CA' which would mean 'California'"
                },
                "unit": {
                    "type": "string",
                    "description": "The unit to fetch the temperature in",
                    "enum": ["celsius", "fahrenheit"]
                }
            },
            "required": ["city", "state", "unit"]
        }
    }
},{
    "type": "function",
    "function": {
        "name": "predict_x",
        "description": "predict x using wd, sd, injection and module_id.",
        "parameters": {
            "type": "object",
            "properties": {
                "wd": {
                    "type":
                    "number",
                    "description":
                    "shi du"
                },
                "sd": {
                    "type":
                    "number",
                    "description":
                    "wen du"
                },
                "injection": {
                    "type":
                    "string",
                    "description":
                    "injection type"
                },
                "module_id": {
                    "type":
                    "string",
                    "description":
                    "module id"
                }
            },
            "required": ["wd", "sd", "injection", "module_id"]
        }
    }
}]

messages = [{
    "role": "user",
    "content": "Hi! How are you doing today?"
}, {
    "role": "assistant",
    "content": "I'm doing well! How can I help you?"
}, {
    "role":
    "user",
    "content":
    "when wd=23, sd=0.8, injection=ZSJB39, module_id=M62121G9142, what's x ? "
}]

chat_completion = client.chat.completions.create(messages=messages,
                                                 model=model,
                                                 tools=tools)

print(chat_completion)
print("\n\n")

tool_calls_stream = client.chat.completions.create(messages=messages,
                                                   model=model,
                                                   tools=tools,
                                                   stream=True)

chunks = []
for chunk in tool_calls_stream:
    chunks.append(chunk)
    if chunk.choices[0].delta.tool_calls:
        arg=chunk.choices[0].delta.tool_calls[0].function.arguments            
        print(f"===={arg}")
    else:
        # print(chunk.choices[0].delta)
        print(f"====❌")

arguments = []
tool_call_idx = -1
for chunk in chunks:
    if chunk.choices[0].delta.tool_calls:
        tool_call = chunk.choices[0].delta.tool_calls[0]

        if tool_call.index != tool_call_idx:
            if tool_call_idx >= 0:
                print(
                    f"streamed tool call arguments: {arguments[tool_call_idx]}"
                )
            tool_call_idx = chunk.choices[0].delta.tool_calls[0].index
            arguments.append("")
        if tool_call.id:
            print(f"streamed tool call id: {tool_call.id} ")

        if tool_call.function:
            if tool_call.function.name:
                print(f"streamed tool call name: {tool_call.function.name}")

            if tool_call.function.arguments:
                arguments[tool_call_idx] += tool_call.function.arguments

if len(arguments)<0:
    print(f"❌no tool used")
else:
    try:
        json.loads(arguments[-1])
        print(f"✅streamed tool call arguments is OK: {arguments[-1]}")
    except:
        print(f"❌streamed tool call arguments is ERROR: {arguments[-1]}")
        

Client log

python openai_chat_completion_client_with_tools_err.py 
model='Qwen2.5-72B-Instruct-AWQ'

ChoiceDelta(content='', function_call=None, refusal=None, role='assistant', tool_calls=None)
stream arg: [None]
stream arg: [{"wd": 2]
stream arg: [3]
stream arg: []
stream arg: []
stream arg: []
stream arg: []
stream arg: []
stream arg: [, "sd": 0]
stream arg: []
stream arg: [.8]
stream arg: []
stream arg: []
stream arg: []
stream arg: []
stream arg: []
stream arg: [, "injection": ""]   <-------------------- 🐛 ERROR here
stream arg: [Z]
stream arg: [S]
stream arg: [JB]
stream arg: [3]
stream arg: [9]
stream arg: []
stream arg: []
stream arg: []
stream arg: []
stream arg: []
stream arg: [", "module_id": "]
stream arg: [M]
stream arg: [6]
stream arg: [2]
stream arg: [1]
stream arg: [2]
stream arg: [1]
stream arg: [G]
stream arg: [9]
stream arg: [1]
stream arg: [4]
stream arg: [2]
stream arg: []
stream arg: [{"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G9142"}]
ChoiceDelta(content='', function_call=None, refusal=None, role=None, tool_calls=None)
streamed tool call id: chatcmpl-tool-4a6872656dae4b278f8754222e16ae64 
streamed tool call name: predict_x
❌streamed tool call arguments is ERROR🐛: {"wd": 23, "sd": 0.8, "injection": ""ZSJB39", "module_id": "M62121G9142{"wd": 23, "sd": 0.8, "injection": "ZSJB39", "module_id": "M62121G9142"}  

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.