4_K_M might be broken...
It's the only quant I tried... but it generates gibrish or nothing... But, Unloth's version of 4_K_M works perfectly....
But, Thank you Bartowski ❤️ TOnsssss ❤️
Hmmm hopefully it's something on your end..! I'll check when I'm back home, sorry about that :(
are you adding a BOS token?
If you are, you should remove it
When I had <|endoftext|> at the start, I also got gibberish
When I removed it, it generated perfectly!
Hope this helps :)
I had the same issue with Q8. Just generated repeated gibberish. Tried with and without BOS.
This is with the latest commit of koboldcpp.
Hmm maybe something off on their end then, works fine in llamacpp :S
Hmm maybe something off on their end then, works fine in llamacpp :S
Yep. Just built llama.cpp and it works. I'll report it to koboldcpp