| tags: | |
| - finetuned | |
| - quantized | |
| - 4-bit | |
| - AWQ | |
| - transformers | |
| - pytorch | |
| - safetensors | |
| - mistral | |
| - text-generation | |
| - finetuned | |
| - conversational | |
| - arxiv:2310.06825 | |
| - license:apache-2.0 | |
| - autotrain_compatible | |
| - has_space | |
| - text-generation-inference | |
| - region:us | |
| model_name: Mistral-7B-Instruct-v0.1-AWQ | |
| base_model: mistralai/Mistral-7B-Instruct-v0.1 | |
| inference: false | |
| model_creator: mistralai | |
| pipeline_tag: text-generation | |
| quantized_by: MaziyarPanahi | |
| # Description | |
| [MaziyarPanahi/Mistral-7B-Instruct-v0.1-AWQ](https://huggingface.co/MaziyarPanahi/Mistral-7B-Instruct-v0.1-AWQ) is a quantized (AWQ) version of [mistralai/Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | |
| ## How to use | |
| ### Install the necessary packages | |
| ``` | |
| pip install --upgrade accelerate autoawq transformers | |
| ``` | |
| ### Example Python code | |
| ```python | |
| from transformers import AutoTokenizer, AutoModelForCausalLM | |
| model_id = "MaziyarPanahi/Mistral-7B-Instruct-v0.1-AWQ" | |
| tokenizer = AutoTokenizer.from_pretrained(model_id) | |
| model = AutoModelForCausalLM.from_pretrained(model_id).to(0) | |
| text = "User:\nHello can you provide me with top-3 cool places to visit in Paris?\n\nAssistant:\n" | |
| inputs = tokenizer(text, return_tensors="pt").to(0) | |
| out = model.generate(**inputs, max_new_tokens=300) | |
| print(tokenizer.decode(out[0], skip_special_tokens=True)) | |
| ``` |