RedHatAI

/

Mistral-Small-3.1-24B-Instruct-2503-quantized.w8a8

like 5

Follow

Red Hat AI 1.14k

Hi @zacksiri . Although this is a mistral model, it is derived from the HuggingFace definition (in contrast to using the original Mistral definition). Hence, you SHOULD NOT use these arguments: --tokenizer_mode mistral --config_format mistral --load_format mistral --tool-call-parser mistral

Also, I noticed that it works on vllm==0.8.3 but it fails on 0.8.4. I'll notify the vLLM team on this

Thank you!

But tool calls aren't getting parsed with vLLM server (without using tokenizer_mode=mistral and other mistral specific arguments) @zacksiri

@cloudadmin-ekstep-vlab i think you can pass the --tool-call-parser and --tokenizer_mode