mradermacher/Kronos-703B-GGUF

About

static quants of https://huggingface.co/Replete-AI/Kronos-703B

Usage

If you are unsure how to use GGUF files, refer to one of TheBloke's READMEs for more details, including on how to concatenate multi-part files.

Provided Quants

(sorted by size, not necessarily quality. IQ-quants are often preferable over similar sized non-IQ quants)

Link	Type	Size/GB	Notes
P1 P2 P3 P4 P5 P6	Q2_K	267.5
P1 P2 P3 P4 P5 P6	IQ3_XS	289.4
P1 P2 P3 P4 P5 P6 P7	IQ3_S	303.1	beats Q3_K*
P1 P2 P3 P4 P5 P6 P7	Q3_K_S	303.1
P1 P2 P3 P4 P5 P6 P7	IQ3_M	326.4
P1 P2 P3 P4 P5 P6 P7 P8	Q3_K_M	353.4	lower quality
P1 P2 P3 P4 P5 P6 P7 P8	IQ4_XS	383.6
P1 P2 P3 P4 P5 P6 P7 P8 P9	Q3_K_L	393.6
P1 P2 P3 P4 P5 P6 P7 P8 P9	IQ4_NL	403.7	prefer IQ4_XS
P1 P2 P3 P4 P5 P6 P7 P8 P9	Q4_K_S	403.7	fast, recommended
P1 P2 P3 P4 P5 P6 P7 P8 P9	Q4_K_M	434.1	fast, recommended
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11	Q5_K_S	484.2
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11	Q5_K_M	503.6
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12	Q6_K	577.4	very good quality
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17	Q8_0	747.5	fast, best quality
P1 P2 P3 P4 P5 P6 P7 P8 P9 P10 P11 P12 P13 P14 P15 P16 P17 P18 P19 P20 P21 P22 P23 P24 P25 P26 P27 P28 P29 P30 P31 P32	SOURCE	1405.9	source gguf, only provided when it was hard to come by

Here is a handy graph by ikawrakow comparing some lower-quality quant types (lower is better):

And here are Artefact2's thoughts on the matter: https://gist.github.com/Artefact2/b5f810600771265fc1e39442288e8ec9

FAQ / Model Request

See https://huggingface.co/mradermacher/model_requests for some answers to questions you might have and/or if you want some other model quantized.

Thanks

I thank my company, nethype GmbH, for letting me use its servers and providing upgrades to my workstation to enable this work in my free time.