meituan

/

DeepSeek-R1-Block-INT8

like 44

Follow

meituan 125

Use the following command to convert DeepSeek-V3-0324-bf16 to int8.
Using sglang for serve, the response is garbled.

python3 bf16_cast_block_int8.py
--input-bf16-hf-path /path/to/deepseek-v3-0324-bf16-weights/
--output-int8-hf-path /path/to/deepseek-v3-0324-int8-weight/