Create README.md
Browse files
README.md
ADDED
@@ -0,0 +1,187 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
+
---
|
2 |
+
tags:
|
3 |
+
- fp8
|
4 |
+
- vllm
|
5 |
+
license: mit
|
6 |
+
license_link: https://huggingface.co/microsoft/Phi-3.5-mini-instruct/resolve/main/LICENSE
|
7 |
+
---
|
8 |
+
|
9 |
+
# Phi-3.5-mini-instruct-FP8-KV
|
10 |
+
|
11 |
+
## Model Overview
|
12 |
+
- **Model Architecture:** Phi-3.5
|
13 |
+
- **Input:** Text
|
14 |
+
- **Output:** Text
|
15 |
+
- **Model Optimizations:**
|
16 |
+
- **Weight quantization:** FP8
|
17 |
+
- **Activation quantization:** FP8
|
18 |
+
- **Intended Use Cases:** Intended for commercial and research use in English. Similarly to [Meta-Llama-3-8B-Instruct](https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct), this models is intended for assistant-like chat.
|
19 |
+
- **Out-of-scope:** Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
|
20 |
+
- **Release Date:** 8/11/2024
|
21 |
+
- **Version:** 1.1
|
22 |
+
- **License(s):** [mit](https://huggingface.co/microsoft/Phi-3.5-mini-instruct/resolve/main/LICENSE)
|
23 |
+
- **Model Developers:** Neural Magic
|
24 |
+
|
25 |
+
Quantized version of [Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct), with the new configuration files.
|
26 |
+
|
27 |
+
### Model Optimizations
|
28 |
+
|
29 |
+
This model was obtained by quantizing the weights and activations of [Phi-3.5-mini-instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct) to FP8 data type, ready for inference with vLLM >= 0.5.1.
|
30 |
+
This optimization reduces the number of bits per parameter from 16 to 8, reducing the disk size and GPU memory requirements by approximately 50%.
|
31 |
+
|
32 |
+
Only the weights and activations of the linear operators within transformers blocks are quantized. Symmetric per-tensor quantization is applied, in which a single linear scaling maps the FP8 representations of the quantized weights and activations.
|
33 |
+
[AutoFP8](https://github.com/neuralmagic/AutoFP8) is used for quantization with 512 sequences of UltraChat.
|
34 |
+
|
35 |
+
## Deployment
|
36 |
+
|
37 |
+
### Use with vLLM
|
38 |
+
|
39 |
+
This model can be deployed efficiently using the [vLLM](https://docs.vllm.ai/en/latest/) backend, as shown in the example below.
|
40 |
+
|
41 |
+
```python
|
42 |
+
from vllm import LLM, SamplingParams
|
43 |
+
from transformers import AutoTokenizer
|
44 |
+
|
45 |
+
model_id = "neuralmagic/Phi-3.5-mini-instruct-FP8-KV"
|
46 |
+
|
47 |
+
sampling_params = SamplingParams(temperature=0.6, top_p=0.9, max_tokens=256)
|
48 |
+
|
49 |
+
tokenizer = AutoTokenizer.from_pretrained(model_id)
|
50 |
+
|
51 |
+
messages = [
|
52 |
+
{"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
|
53 |
+
{"role": "user", "content": "Who are you? Remember to respond in pirate speak!"},
|
54 |
+
]
|
55 |
+
|
56 |
+
prompts = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
57 |
+
|
58 |
+
llm = LLM(model=model_id, kv_cache_dtype="fp8")
|
59 |
+
|
60 |
+
outputs = llm.generate(prompts, sampling_params)
|
61 |
+
|
62 |
+
generated_text = outputs[0].outputs[0].text
|
63 |
+
print(generated_text)
|
64 |
+
```
|
65 |
+
|
66 |
+
vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
|
67 |
+
|
68 |
+
## Creation
|
69 |
+
|
70 |
+
This model was created by applying [LLM Compressor with calibration samples from UltraChat](https://github.com/vllm-project/llm-compressor/blob/sa/big_model_support/examples/big_model_offloading/big_model_w8a8_calibrate.py), as presented in the code snipet below.
|
71 |
+
|
72 |
+
```python
|
73 |
+
from datasets import load_dataset
|
74 |
+
from transformers import AutoTokenizer
|
75 |
+
|
76 |
+
from llmcompressor.transformers import SparseAutoModelForCausalLM, oneshot
|
77 |
+
|
78 |
+
# Select model and load it.
|
79 |
+
# Phi-3.5 is a special case for KV cache quantization because it has
|
80 |
+
# fused QKV linear layers.
|
81 |
+
MODEL_ID = "microsoft/Phi-3.5-mini-instruct"
|
82 |
+
model = SparseAutoModelForCausalLM.from_pretrained(
|
83 |
+
MODEL_ID,
|
84 |
+
device_map="auto",
|
85 |
+
torch_dtype="auto",
|
86 |
+
)
|
87 |
+
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID)
|
88 |
+
|
89 |
+
# Select calibration dataset.
|
90 |
+
DATASET_ID = "HuggingFaceH4/ultrachat_200k"
|
91 |
+
DATASET_SPLIT = "train_sft"
|
92 |
+
|
93 |
+
# Select number of samples. 512 samples is a good place to start.
|
94 |
+
# Increasing the number of samples can improve accuracy.
|
95 |
+
NUM_CALIBRATION_SAMPLES = 512
|
96 |
+
MAX_SEQUENCE_LENGTH = 2048
|
97 |
+
|
98 |
+
# Load dataset and preprocess.
|
99 |
+
ds = load_dataset(DATASET_ID, split=DATASET_SPLIT)
|
100 |
+
ds = ds.shuffle(seed=42).select(range(NUM_CALIBRATION_SAMPLES))
|
101 |
+
|
102 |
+
|
103 |
+
def process_and_tokenize(example):
|
104 |
+
text = tokenizer.apply_chat_template(example["messages"], tokenize=False)
|
105 |
+
return tokenizer(
|
106 |
+
text,
|
107 |
+
padding=False,
|
108 |
+
max_length=MAX_SEQUENCE_LENGTH,
|
109 |
+
truncation=True,
|
110 |
+
add_special_tokens=False,
|
111 |
+
)
|
112 |
+
|
113 |
+
|
114 |
+
ds = ds.map(process_and_tokenize, remove_columns=ds.column_names)
|
115 |
+
|
116 |
+
# Configure the quantization algorithm and scheme.
|
117 |
+
# In this case, we:
|
118 |
+
# * quantize the weights to fp8 with per-tensor scales
|
119 |
+
# * quantize the activations to fp8 with per-tensor scales
|
120 |
+
# * quantize the kv cache to fp8 with per-tensor scales
|
121 |
+
recipe = """
|
122 |
+
quant_stage:
|
123 |
+
quant_modifiers:
|
124 |
+
QuantizationModifier:
|
125 |
+
ignore: ["lm_head"]
|
126 |
+
config_groups:
|
127 |
+
group_0:
|
128 |
+
weights:
|
129 |
+
num_bits: 8
|
130 |
+
type: float
|
131 |
+
strategy: tensor
|
132 |
+
dynamic: false
|
133 |
+
symmetric: true
|
134 |
+
input_activations:
|
135 |
+
num_bits: 8
|
136 |
+
type: float
|
137 |
+
strategy: tensor
|
138 |
+
dynamic: false
|
139 |
+
symmetric: true
|
140 |
+
targets: ["Linear"]
|
141 |
+
kv_cache_scheme:
|
142 |
+
num_bits: 8
|
143 |
+
type: float
|
144 |
+
strategy: tensor
|
145 |
+
dynamic: false
|
146 |
+
symmetric: true
|
147 |
+
"""
|
148 |
+
|
149 |
+
# Apply algorithms.
|
150 |
+
oneshot(
|
151 |
+
model=model,
|
152 |
+
dataset=ds,
|
153 |
+
recipe=recipe,
|
154 |
+
max_seq_length=MAX_SEQUENCE_LENGTH,
|
155 |
+
num_calibration_samples=NUM_CALIBRATION_SAMPLES,
|
156 |
+
)
|
157 |
+
|
158 |
+
# Confirm generations of the quantized model look sane.
|
159 |
+
print("\n\n")
|
160 |
+
print("========== SAMPLE GENERATION ==============")
|
161 |
+
input_ids = tokenizer("Hello my name is", return_tensors="pt").input_ids.to("cuda")
|
162 |
+
output = model.generate(input_ids, max_new_tokens=100)
|
163 |
+
print(tokenizer.decode(output[0]))
|
164 |
+
print("==========================================\n\n")
|
165 |
+
|
166 |
+
# Save to disk compressed.
|
167 |
+
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-KV"
|
168 |
+
model.save_pretrained(SAVE_DIR, save_compressed=True)
|
169 |
+
tokenizer.save_pretrained(SAVE_DIR)
|
170 |
+
```
|
171 |
+
|
172 |
+
## Evaluation
|
173 |
+
|
174 |
+
The model was evaluated on the [OpenLLM](https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard) leaderboard tasks (version 1) with the [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness/tree/383bbd54bc621086e05aa1b030d8d4d5635b25e6) (commit 383bbd54bc621086e05aa1b030d8d4d5635b25e6) and the [vLLM](https://docs.vllm.ai/en/stable/) engine, using the following command:
|
175 |
+
```
|
176 |
+
lm_eval \
|
177 |
+
--model vllm \
|
178 |
+
--model_args pretrained="neuralmagic/Phi-3.5-mini-instruct-FP8-KV",kv_cache_dtype="fp8",gpu_memory_utilization=0.4,add_bos_token=True,max_model_len=4096 \
|
179 |
+
--tasks openllm \
|
180 |
+
--batch_size auto
|
181 |
+
```
|
182 |
+
|
183 |
+
### Accuracy
|
184 |
+
|
185 |
+
#### Open LLM Leaderboard evaluation scores
|
186 |
+
|
187 |
+
TBD
|