mradermacher

/

Camel-Doc-OCR-062825-GGUF

like 1

Yes this model does support vision. For vision you need to provide booth the GGUF of the LLM layers quantized in any way you like and the vision layer as mmproj file for which you can use one of the following files:

• https://huggingface.co/mradermacher/Camel-Doc-OCR-062825-GGUF/blob/main/Camel-Doc-OCR-062825.mmproj-Q8_0.gguf
• https://huggingface.co/mradermacher/Camel-Doc-OCR-062825-GGUF/blob/main/Camel-Doc-OCR-062825.mmproj-f16.gguf

If ollama doesn't work just use llama.cpp directly and it will work for sure.

Thanks! Any info how to use that with vllm?

Thanks! Any info how to use that with vllm?

GGUF support in vLLM is experimental and currently doesn't support vision. Generally using GGUFs on vLLM is a bit of a waste. Other quant formats such as AWQ are over 20 times faster with vLLM GPU if you have 64 parallel prompts. If you want to use GGUFs then the way to go is llama.cpp. llama.cpp is the reference implementation of the GGUF format and so usually has the most recent features and the least number of issues. With llama-server llama.cpp overs a nice GUI.

That makes sense, thank you for your help!