Update README.md
Browse files
README.md
CHANGED
@@ -13,7 +13,7 @@ This repository hosts the **Qwen3-8B** model quantized with [torchao](https://hu
|
|
13 |
using int4 weight-only quantization and the [awq](https://arxiv.org/abs/2306.00978) algorithm.
|
14 |
This work is brought to you by the PyTorch team. This model can be used directly or served using [vLLM](https://docs.vllm.ai/en/latest/) for 53% VRAM reduction (7.82 GB needed)
|
15 |
and 1.34x speedup on H100 GPUs for batch size 1. The model is calibrated with 10 samples from `mmlu_abstract_algebra` task to recover the accuracy for `mmlu_abstract_algebra` specifically.
|
16 |
-
AWQ-INT4 improves the accuracy of `mmlu_abstract_algebra` of INT4 from 55 to 56, while the
|
17 |
|
18 |
# Inference with vLLM
|
19 |
Install vllm nightly and torchao nightly to get some recent changes:
|
|
|
13 |
using int4 weight-only quantization and the [awq](https://arxiv.org/abs/2306.00978) algorithm.
|
14 |
This work is brought to you by the PyTorch team. This model can be used directly or served using [vLLM](https://docs.vllm.ai/en/latest/) for 53% VRAM reduction (7.82 GB needed)
|
15 |
and 1.34x speedup on H100 GPUs for batch size 1. The model is calibrated with 10 samples from `mmlu_abstract_algebra` task to recover the accuracy for `mmlu_abstract_algebra` specifically.
|
16 |
+
AWQ-INT4 improves the accuracy of `mmlu_abstract_algebra` of INT4 from 55 to 56, while the bfloat16 baseline is 58.
|
17 |
|
18 |
# Inference with vLLM
|
19 |
Install vllm nightly and torchao nightly to get some recent changes:
|