tool calling with vllm
#1
by
krsnvijay
- opened
docker run --name vllm-model --restart always --runtime nvidia --gpus all -p 8020:8000 --ipc=host -d vllm/vllm-openai:latest --model bullerwins/Devstral-Small-2505-fp8 --tensor-parallel-size 2 --max-model-len 70000 --enable-auto-tool-choice --tool-call-parser mistral --tokenizer_mode mistral
Hi would you know how to get tool calling to work with vllm
These are from vllm logs when i make a tool call, i'm running vllm 0.8.5.post1
INFO 05-21 11:33:58 [chat_utils.py:397] Detected the chat template content format to be 'string'. You can set `--chat-template-content-format` to override this.
WARNING 05-21 11:33:58 [chat_utils.py:318] 'add_generation_prompt' is not supported for mistral tokenizer, so it will be ignored.
WARNING 05-21 11:33:58 [chat_utils.py:322] 'continue_final_message' is not supported for mistral tokenizer, so it will be ignored.
ERROR 05-21 11:33:58 [serving_chat.py:200] Error in preprocessing prompt inputs
ERROR 05-21 11:33:58 [serving_chat.py:200] Traceback (most recent call last):
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/mistral_common/protocol/instruct/validator.py", line 76, in _validate_function
ERROR 05-21 11:33:58 [serving_chat.py:200] Draft7Validator.check_schema(function.parameters)
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/jsonschema/validators.py", line 317, in check_schema
ERROR 05-21 11:33:58 [serving_chat.py:200] raise exceptions.SchemaError.create_from(error)
ERROR 05-21 11:33:58 [serving_chat.py:200] jsonschema.exceptions.SchemaError: 'list of integer or string' is not valid under any of the given schemas
ERROR 05-21 11:33:58 [serving_chat.py:200]
ERROR 05-21 11:33:58 [serving_chat.py:200] Failed validating 'anyOf' in metaschema['properties']['properties']['additionalProperties']['properties']['type']:
ERROR 05-21 11:33:58 [serving_chat.py:200] {'anyOf': [{'$ref': '#/definitions/simpleTypes'},
ERROR 05-21 11:33:58 [serving_chat.py:200] {'type': 'array',
ERROR 05-21 11:33:58 [serving_chat.py:200] 'items': {'$ref': '#/definitions/simpleTypes'},
ERROR 05-21 11:33:58 [serving_chat.py:200] 'minItems': 1,
ERROR 05-21 11:33:58 [serving_chat.py:200] 'uniqueItems': True}]}
ERROR 05-21 11:33:58 [serving_chat.py:200]
ERROR 05-21 11:33:58 [serving_chat.py:200] On schema['properties']['uids']['type']:
ERROR 05-21 11:33:58 [serving_chat.py:200] 'list of integer or string'
ERROR 05-21 11:33:58 [serving_chat.py:200]
ERROR 05-21 11:33:58 [serving_chat.py:200] During handling of the above exception, another exception occurred:
ERROR 05-21 11:33:58 [serving_chat.py:200]
ERROR 05-21 11:33:58 [serving_chat.py:200] Traceback (most recent call last):
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 1238, in apply_mistral_chat_template
ERROR 05-21 11:33:58 [serving_chat.py:200] return tokenizer.apply_chat_template(
ERROR 05-21 11:33:58 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/vllm/transformers_utils/tokenizers/mistral.py", line 379, in apply_chat_template
ERROR 05-21 11:33:58 [serving_chat.py:200] encoded = self.mistral.encode_chat_completion(request)
ERROR 05-21 11:33:58 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/mistral_common/tokens/tokenizers/mistral.py", line 230, in encode_chat_completion
ERROR 05-21 11:33:58 [serving_chat.py:200] validated_request = self._chat_completion_request_validator.validate_request(request)
ERROR 05-21 11:33:58 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/mistral_common/protocol/instruct/validator.py", line 66, in validate_request
ERROR 05-21 11:33:58 [serving_chat.py:200] self._validate_tools(request.tools or [])
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/mistral_common/protocol/instruct/validator.py", line 93, in _validate_tools
ERROR 05-21 11:33:58 [serving_chat.py:200] self._validate_function(tool.function)
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/mistral_common/protocol/instruct/validator.py", line 78, in _validate_function
ERROR 05-21 11:33:58 [serving_chat.py:200] raise InvalidToolSchemaException(f"Invalid tool schema: {e.message}")
ERROR 05-21 11:33:58 [serving_chat.py:200] mistral_common.exceptions.InvalidToolSchemaException: Invalid tool schema: 'list of integer or string' is not valid under any of the given schemas
ERROR 05-21 11:33:58 [serving_chat.py:200]
ERROR 05-21 11:33:58 [serving_chat.py:200] The above exception was the direct cause of the following exception:
ERROR 05-21 11:33:58 [serving_chat.py:200]
ERROR 05-21 11:33:58 [serving_chat.py:200] Traceback (most recent call last):
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 183, in create_chat_completion
ERROR 05-21 11:33:58 [serving_chat.py:200] ) = await self._preprocess_chat(
ERROR 05-21 11:33:58 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 421, in _preprocess_chat
ERROR 05-21 11:33:58 [serving_chat.py:200] request_prompt = apply_mistral_chat_template(
ERROR 05-21 11:33:58 [serving_chat.py:200] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-21 11:33:58 [serving_chat.py:200] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/chat_utils.py", line 1248, in apply_mistral_chat_template
ERROR 05-21 11:33:58 [serving_chat.py:200] raise ValueError from e
ERROR 05-21 11:33:58 [serving_chat.py:200] ValueError
INFO: 10.0.0.249:47748 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request