jerryzh168 commited on
Commit
37df94d
·
verified ·
1 Parent(s): f808b58

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -16,7 +16,7 @@ base_model:
16
  ---
17
 
18
  [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) quantized with [torchao](https://huggingface.co/docs/transformers/main/en/quantization/torchao) int4 weight only quantization, using [hqq](https://mobiusml.github.io/hqq_blog/) algorithm for improved accuracy, by PyTorch team.
19
- Use it directly or serve using [vLLM](https://docs.vllm.ai/en/latest/) for 62% VRAM reduction and 20%+ speedup on A100 GPUs.
20
 
21
  # Inference with vLLM
22
  Install vllm nightly and torchao nightly to get some recent changes:
@@ -282,7 +282,7 @@ Our int4wo is only optimized for batch size 1, so expect some slowdown with larg
282
  | Benchmark (Latency) | | |
283
  |----------------------------------|----------------|--------------------------|
284
  | | Qwen3-8B | Qwen3-8B-int4wo-hqq |
285
- | latency (batch_size=1) | 3.52s | 2.84s (24% speedup) |
286
 
287
  Int4 weight only is optimized for batch size 1 and short input and output token length, please stay tuned for models optimized for larger batch sizes or longer token length.
288
 
 
16
  ---
17
 
18
  [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B) quantized with [torchao](https://huggingface.co/docs/transformers/main/en/quantization/torchao) int4 weight only quantization, using [hqq](https://mobiusml.github.io/hqq_blog/) algorithm for improved accuracy, by PyTorch team.
19
+ Use it directly or serve using [vLLM](https://docs.vllm.ai/en/latest/) for 62% VRAM reduction and 1.2x speedup on A100 GPUs.
20
 
21
  # Inference with vLLM
22
  Install vllm nightly and torchao nightly to get some recent changes:
 
282
  | Benchmark (Latency) | | |
283
  |----------------------------------|----------------|--------------------------|
284
  | | Qwen3-8B | Qwen3-8B-int4wo-hqq |
285
+ | latency (batch_size=1) | 3.52s | 2.84s (1.24x speedup) |
286
 
287
  Int4 weight only is optimized for batch size 1 and short input and output token length, please stay tuned for models optimized for larger batch sizes or longer token length.
288