gguf q4 version ?
#3
by
exclusif20
- opened
Hello, are you going to deploy a gguf q4 version ?
I'm also waiting. GGUF is a very popular format since it has many GUIs and tools and very easy to run/use/deploy. I wonder when ONNX will have such software. I mean a single exe that can do a lot like KOBOLDCPP(text, visual, stt, tts, even image generation). starting compatible servers for these would do.
if ONNX ever gets software like that (with option for OpenVino Backend like sd.next), I would definitely try it since it is more capable of having all kinds of models(even non llm/sd).