File size: 2,224 Bytes
a546bf4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
---
license: apache-2.0
base_model: Qwen/Qwen3-4B
base_model_relation: quantized
tags:
- Qwen
- Qwen3
- GGUF
- quantized
- 8-bit
---
## Llama.cpp hybrid layer quantization of Qwen3-4B by Alibaba
Original model: https://huggingface.co/Qwen/Qwen3-4B
The hybrid quant employs different quantization levels on a per layer basis to increased
flexibility of trading off performance vs file size. Less parameter bits are used at deep layers
and more bits at cortex layers to simultaneously optimize quantized size and model performance.
These quants were specifically optimized for the Qwen3 4B edge model for essentially no performance loss
vs Q8_0 quant while reducing file size about 0.6G.
The layer quants are as follows:
```
LAYER_TYPES='[
[0 ,"Q8_0" ],[1 ,"Q5_K_M"],[2 ,"Q5_K_M"],[3 ,"Q5_K_M"],[4 ,"Q5_K_M"],[5 ,"Q5_K_M"],
[6 ,"Q5_K_M"],[7 ,"Q5_K_M"],[8, "Q5_K_M"],[9, "Q5_K_M"],[10,"Q5_K_M"],[11,"Q5_K_M"],
[12,"Q6_K" ],[13,"Q6_K" ],[14,"Q6_K" ],[15,"Q6_K" ],[16,"Q6_K" ],[17,"Q6_K" ],
[18,"Q6_K" ],[19,"Q6_K" ],[20,"Q6_K" ],[21,"Q6_K" ],[22,"Q6_K" ],[23,"Q6_K" ],
[24,"Q8_0" ],[25,"Q8_0" ],[26,"Q8_0" ],[27,"Q8_0" ],[28,"Q8_0" ],[29,"Q8_0" ],
[30,"Q8_0" ],[31,"Q8_0" ],[32,"Q8_0" ],[33,"Q8_0" ],[34,"Q8_0" ],[35,"Q8_0" ]
]'
FLAGS="--token-embedding-type Q8_0 --output-tensor-type Q6_K"
```
These quants were select based on combined subjective and objective performance
evaluations to give both high performance and reduced file size.
Comparison:
Quant | size | PPL | Comment
---------|---------|------|-----------
Q8_0 | 4.3e9 | 13.2 | default embed and output
Q8_0_H | 3.6e9 | 13.1 | Q8_0 embed Q6_K output
Full evals comparing Qwen3 4B Q8_0 and Q8_0_H are also available at https://huggingface.co/spaces/steampunque/benchlm
## Download the file from below:
| Link | Type | Size/e9 B | Notes |
|------|------|-----------|-------|
| [Qwen3-4B.Q8_0_H.gguf](https://huggingface.co/steampunque/Qwen3-4B-Hybrid-GGUF/resolve/main/Qwen3-4B.Q8_0_H.gguf) | Q8_0_H | 3.6e9 B | Q8_0 quality |
A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:
https://github.com/ggml-org/llama.cpp/discussions/13040 |