File size: 1,351 Bytes
1331ea5 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 |
---
tags:
- finetuned
- quantized
- 4-bit
- AWQ
- transformers
- pytorch
- safetensors
- mistral
- text-generation
- finetuned
- conversational
- arxiv:2310.06825
- license:apache-2.0
- autotrain_compatible
- has_space
- text-generation-inference
- region:us
model_name: Mistral-7B-Instruct-v0.1-AWQ
base_model: mistralai/Mistral-7B-Instruct-v0.1
inference: false
model_creator: mistralai
pipeline_tag: text-generation
quantized_by: MaziyarPanahi
---
# Description
[MaziyarPanahi/Mistral-7B-Instruct-v0.1-AWQ](https://huggingface.co/MaziyarPanahi/Mistral-7B-Instruct-v0.1-AWQ) is a quantized (AWQ) version of [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1)
## How to use
### Install the necessary packages
```
pip install --upgrade accelerate autoawq transformers
```
### Example Python code
```python
from transformers import AutoTokenizer, AutoModelForCausalLM
model_id = "MaziyarPanahi/Mistral-7B-Instruct-v0.1-AWQ"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id).to(0)
text = "User:\nHello can you provide me with top-3 cool places to visit in Paris?\n\nAssistant:\n"
inputs = tokenizer(text, return_tensors="pt").to(0)
out = model.generate(**inputs, max_new_tokens=300)
print(tokenizer.decode(out[0], skip_special_tokens=True))
``` |