IQ4_NL vs IQ4_XS vs Q4_K_S vs Q4_0
#5
by
CosmossG
- opened
What is the difference between IQ4_NL(?), IQ4_XS (eXtra Small?), Q4_K_S and Q4_0? I understand how Imatrix quants are generally better than normal quants, but I am a little confused on the NL quant. Could you please clarify that for me? Thanks in advance!
Also, are these models better than the base GGUF's (e.g UnSloth's GGUF's) on these low quants for math/logic and general "smartness"? Also, as far as I know, these GGUF's also do not come with vision, right?
RE: NL
Non-linear ; this is an oddball quant which I find (like IQ4XS) good for creative use cases.
This is also used in the case of odd layers/sub-layers in a model which do not conform to standard sizes.
RE: Unsloth.
They should be the same.
Correct ; no vision.