jerryzh168 commited on
Commit
ae7fb3c
·
verified ·
1 Parent(s): abbab01

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -1
README.md CHANGED
@@ -13,7 +13,7 @@ This repository hosts the **Qwen3-8B** model quantized with [torchao](https://hu
13
  using int4 weight-only quantization and the [awq](https://arxiv.org/abs/2306.00978) algorithm.
14
  This work is brought to you by the PyTorch team. This model can be used directly or served using [vLLM](https://docs.vllm.ai/en/latest/) for 53% VRAM reduction (7.82 GB needed)
15
  and 1.34x speedup on H100 GPUs for batch size 1. The model is calibrated with 10 samples from `mmlu_abstract_algebra` task to recover the accuracy for `mmlu_abstract_algebra` specifically.
16
- AWQ-INT4 improves the accuracy of `mmlu_abstract_algebra` of INT4 from 55 to 56, while the non quantized baseline is 58.
17
 
18
  # Inference with vLLM
19
  Install vllm nightly and torchao nightly to get some recent changes:
 
13
  using int4 weight-only quantization and the [awq](https://arxiv.org/abs/2306.00978) algorithm.
14
  This work is brought to you by the PyTorch team. This model can be used directly or served using [vLLM](https://docs.vllm.ai/en/latest/) for 53% VRAM reduction (7.82 GB needed)
15
  and 1.34x speedup on H100 GPUs for batch size 1. The model is calibrated with 10 samples from `mmlu_abstract_algebra` task to recover the accuracy for `mmlu_abstract_algebra` specifically.
16
+ AWQ-INT4 improves the accuracy of `mmlu_abstract_algebra` of INT4 from 55 to 56, while the bfloat16 baseline is 58.
17
 
18
  # Inference with vLLM
19
  Install vllm nightly and torchao nightly to get some recent changes: