What's the quantization format of 4bit / 8bit?
#39
by
WatermelonEast
- opened
or it means fp4 / fp8?
Hi @WatermelonEast ,
The quantization format for 8-bit precision is int8
, and for 4-bit precision, it is int4
. To enable these quantization formats, you can use the following lines of code:
quantization_config = BitsAndBytesConfig(load_in_8bit=True)
quantization_config = BitsAndBytesConfig(load_in_4bit=True)
Thank you.