Update README.md
Browse files
README.md
CHANGED
@@ -13,9 +13,9 @@ tags:
|
|
13 |
- gptq
|
14 |
- intel
|
15 |
license: apache-2.0
|
16 |
-
model_name: Minerva
|
17 |
base_model:
|
18 |
-
- sapienzanlp/Minerva-
|
19 |
inference: false
|
20 |
model_creator: sapienzanlp
|
21 |
datasets:
|
@@ -28,7 +28,7 @@ quantized_by: fbaldassarri
|
|
28 |
|
29 |
## Model Information
|
30 |
|
31 |
-
Quantized version of [sapienzanlp/Minerva-
|
32 |
- 8 bits (INT8)
|
33 |
- group size = 128
|
34 |
- Symmetrical Quantization
|
@@ -38,7 +38,7 @@ Fast and low memory, 2-3X speedup (slight accuracy drop at W8G128)
|
|
38 |
|
39 |
Quantization framework: [Intel AutoRound](https://github.com/intel/auto-round) v0.4.6
|
40 |
|
41 |
-
Note: this INT8 version of Minerva-
|
42 |
|
43 |
## Replication Recipe
|
44 |
|
@@ -63,14 +63,14 @@ pip install -vvv --no-build-isolation -e .[cpu]
|
|
63 |
|
64 |
```
|
65 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
66 |
-
model_name = "sapienzanlp/Minerva-
|
67 |
model = AutoModelForCausalLM.from_pretrained(model_name)
|
68 |
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
69 |
from auto_round import AutoRound
|
70 |
bits, group_size, sym, device, amp = 8, 128, True, 'cpu', False
|
71 |
autoround = AutoRound(model, tokenizer, nsamples=128, iters=200, seqlen=512, batch_size=4, bits=bits, group_size=group_size, sym=sym, device=device, amp=amp)
|
72 |
autoround.quantize()
|
73 |
-
output_dir = "./AutoRound/sapienzanlp_Minerva-
|
74 |
autoround.save_quantized(output_dir, format='auto_round', inplace=True)
|
75 |
```
|
76 |
|
|
|
13 |
- gptq
|
14 |
- intel
|
15 |
license: apache-2.0
|
16 |
+
model_name: Minerva 350M base v1.0
|
17 |
base_model:
|
18 |
+
- sapienzanlp/Minerva-350M-base-v1.0
|
19 |
inference: false
|
20 |
model_creator: sapienzanlp
|
21 |
datasets:
|
|
|
28 |
|
29 |
## Model Information
|
30 |
|
31 |
+
Quantized version of [sapienzanlp/Minerva-350M-base-v1.0](https://huggingface.co/sapienzanlp/Minerva-350M-base-v1.0) using torch.float32 for quantization tuning.
|
32 |
- 8 bits (INT8)
|
33 |
- group size = 128
|
34 |
- Symmetrical Quantization
|
|
|
38 |
|
39 |
Quantization framework: [Intel AutoRound](https://github.com/intel/auto-round) v0.4.6
|
40 |
|
41 |
+
Note: this INT8 version of Minerva-350M-base-v1.0 has been quantized to run inference through CPU.
|
42 |
|
43 |
## Replication Recipe
|
44 |
|
|
|
63 |
|
64 |
```
|
65 |
from transformers import AutoModelForCausalLM, AutoTokenizer
|
66 |
+
model_name = "sapienzanlp/Minerva-350M-base-v1.0"
|
67 |
model = AutoModelForCausalLM.from_pretrained(model_name)
|
68 |
tokenizer = AutoTokenizer.from_pretrained(model_name)
|
69 |
from auto_round import AutoRound
|
70 |
bits, group_size, sym, device, amp = 8, 128, True, 'cpu', False
|
71 |
autoround = AutoRound(model, tokenizer, nsamples=128, iters=200, seqlen=512, batch_size=4, bits=bits, group_size=group_size, sym=sym, device=device, amp=amp)
|
72 |
autoround.quantize()
|
73 |
+
output_dir = "./AutoRound/sapienzanlp_Minerva-350M-base-v1.0-autoround-int8-gs128-sym"
|
74 |
autoround.save_quantized(output_dir, format='auto_round', inplace=True)
|
75 |
```
|
76 |
|