Update README.md
Browse files
README.md
CHANGED
@@ -16,7 +16,7 @@ base_model:
|
|
16 |
---
|
17 |
|
18 |
[Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) quantized with [torchao](https://huggingface.co/docs/transformers/main/en/quantization/torchao) int4 weight only quantization, using [hqq](https://mobiusml.github.io/hqq_blog/) algorithm for improved accuracy, by PyTorch team.
|
19 |
-
Use it directly or serve using [vLLM](https://docs.vllm.ai/en/latest/) for 62% VRAM reduction and
|
20 |
|
21 |
# Inference with vLLM
|
22 |
Install vllm nightly and torchao nightly to get some recent changes:
|
@@ -282,7 +282,7 @@ Our int4wo is only optimized for batch size 1, so expect some slowdown with larg
|
|
282 |
| Benchmark (Latency) | | |
|
283 |
|----------------------------------|----------------|--------------------------|
|
284 |
| | Qwen3-8B | Qwen3-8B-int4wo-hqq |
|
285 |
-
| latency (batch_size=1) | 3.52s | 2.84s (
|
286 |
|
287 |
Int4 weight only is optimized for batch size 1 and short input and output token length, please stay tuned for models optimized for larger batch sizes or longer token length.
|
288 |
|
|
|
16 |
---
|
17 |
|
18 |
[Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) quantized with [torchao](https://huggingface.co/docs/transformers/main/en/quantization/torchao) int4 weight only quantization, using [hqq](https://mobiusml.github.io/hqq_blog/) algorithm for improved accuracy, by PyTorch team.
|
19 |
+
Use it directly or serve using [vLLM](https://docs.vllm.ai/en/latest/) for 62% VRAM reduction and 1.2x speedup on A100 GPUs.
|
20 |
|
21 |
# Inference with vLLM
|
22 |
Install vllm nightly and torchao nightly to get some recent changes:
|
|
|
282 |
| Benchmark (Latency) | | |
|
283 |
|----------------------------------|----------------|--------------------------|
|
284 |
| | Qwen3-8B | Qwen3-8B-int4wo-hqq |
|
285 |
+
| latency (batch_size=1) | 3.52s | 2.84s (1.24x speedup) |
|
286 |
|
287 |
Int4 weight only is optimized for batch size 1 and short input and output token length, please stay tuned for models optimized for larger batch sizes or longer token length.
|
288 |
|