How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study
Paper
•
2404.14047
•
Published
•
45
•
12
wenhua cheng
wenhuach
AI & ML interests
Model Compression, CV
Recent Activity
new activity
1 day ago
kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-4bit:how to run this model
reacted
to
their
post
with 🚀
26 days ago
Check out [DeepSeek-R1 INT2 model(https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc). This 200GB DeepSeek-R1 model shows only about a 2% drop in MMLU, though it's quite slow due to kernel issue.
| | BF16 | INT2-mixed |
| ------------- | ------ | ---------- |
| mmlu | 0.8514 | 0.8302 |
| hellaswag | 0.6935 | 0.6657 |
| winogrande | 0.7932 | 0.7940 |
| arc_challenge | 0.6212 | 0.6084 |
posted
an
update
26 days ago
Check out [DeepSeek-R1 INT2 model(https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc). This 200GB DeepSeek-R1 model shows only about a 2% drop in MMLU, though it's quite slow due to kernel issue.
| | BF16 | INT2-mixed |
| ------------- | ------ | ---------- |
| mmlu | 0.8514 | 0.8302 |
| hellaswag | 0.6935 | 0.6657 |
| winogrande | 0.7932 | 0.7940 |
| arc_challenge | 0.6212 | 0.6084 |
Organizations
wenhuach's activity
how to run this model
4
#1 opened 1 day ago
by
cicdatopea
Update README.md
#1 opened 4 months ago
by
wenhuach
AutoRound now supports full range sym
#1 opened 5 months ago
by
wenhuach
Model files missing?
3
#1 opened 8 months ago
by
akashaero
Add support for AQLM
3
#1 opened 11 months ago
by
BlackSamorez
Introducing AutoRound INT4 algorithm
4
#4 opened 11 months ago
by
wenhuach
commented
a
paper
11 months ago
Introducing AutoRound int4 algoirhtm
#14 opened 12 months ago
by
wenhuach
Introducing AutoRound INT4 Algorithm
#12 opened about 1 year ago
by
wenhuach
commented
a
paper
about 1 year ago