Adding model card

README.md

@@ -1,3 +1,115 @@
 ---
-license: apache-2.0
+language:
+- en
+license: cc-by-nc-4.0
+tags:
+- text-generation-inference
+- transformers
+- quantized
+- 4-bit
+- AWQ
+- text-generation
+- autotrain_compatible
+- endpoints_compatible
+- chatml
+- mistral
+model_creator: SanjiWatsuki
+model_name: Kunoichi-DPO-v2-7B
+model_type: mistral
+pipeline_tag: text-generation
+inference: false
 ---
+# SanjiWatsuki/Kunoichi-DPO-v2-7B AWQ
+
+**UPLOAD IN PROGRESS**
+
+- Model creator: [SanjiWatsuki](https://huggingface.co/SanjiWatsuki)
+- Original model: [Kunoichi-DPO-v2-7B](https://huggingface.co/macadeliccc/Kunoichi-DPO-v2-7B)
+
+## Model Summary
+
+| Model | Average | AGIEval | GPT4All | TruthfulQA | Bigbench |
+|---|---:|---:|---:|---:|---:|
+| **Kunoichi-DPO-7B**|**58.4**|  45.08 |  74|     66.99|   47.52|
+| **Kunoichi-DPO-v2-7B**|**58.31**|  44.85|  75.05|     65.69|   47.65|
+| [Kunoichi-7B](https://huggingface.co/SanjiWatsuki/Kunoichi-7B)|57.54|  44.99|  74.86|     63.72|   46.58|
+| [OpenPipe/mistral-ft-optimized-1218](https://huggingface.co/OpenPipe/mistral-ft-optimized-1218)| 56.85 | 44.74 | 75.6 | 59.89 | 47.17 |
+| [Silicon-Maid-7B](https://huggingface.co/SanjiWatsuki/Silicon-Maid-7B) | 56.45|  44.74|  74.26|      61.5|   45.32|
+| [mlabonne/NeuralHermes-2.5-Mistral-7B](https://huggingface.co/mlabonne/NeuralHermes-2.5-Mistral-7B) | 53.51 | 43.67 | 73.24 | 55.37 | 41.76 |
+| [teknium/OpenHermes-2.5-Mistral-7B](https://huggingface.co/teknium/OpenHermes-2.5-Mistral-7B)  | 52.42 | 42.75 | 72.99 | 52.99 | 40.94 |
+| [openchat/openchat_3.5](https://huggingface.co/openchat/openchat_3.5) | 51.34 | 42.67 | 72.92 | 47.27 | 42.51 |
+| [berkeley-nest/Starling-LM-7B-alpha](https://huggingface.co/berkeley-nest/Starling-LM-7B-alpha) | 51.16 | 42.06 | 72.72 | 47.33 | 42.53 |
+| [HuggingFaceH4/zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) | 50.99 | 37.33 | 71.83 | 55.1 | 39.7 |
+
+## How to use
+
+### Install the necessary packages
+
+```bash
+pip install --upgrade autoawq autoawq-kernels
+```
+
+### Example Python code
+
+```python
+from awq import AutoAWQForCausalLM
+from transformers import AutoTokenizer, TextStreamer
+
+model_path = "solidrust/Kunoichi-DPO-v2-7B-AWQ"
+system_message = "You are Kunoichi, incarnated as a powerful AI."
+
+# Load model
+model = AutoAWQForCausalLM.from_quantized(model_path,
+                                          fuse_layers=True)
+tokenizer = AutoTokenizer.from_pretrained(model_path,
+                                          trust_remote_code=True)
+streamer = TextStreamer(tokenizer,
+                        skip_prompt=True,
+                        skip_special_tokens=True)
+
+# Convert prompt to tokens
+prompt_template = """\
+<|im_start|>system
+{system_message}<|im_end|>
+<|im_start|>user
+{prompt}<|im_end|>
+<|im_start|>assistant"""
+
+prompt = "You're standing on the surface of the Earth. "\
+        "You walk one mile south, one mile west and one mile north. "\
+        "You end up exactly where you started. Where are you?"
+
+tokens = tokenizer(prompt_template.format(system_message=system_message,prompt=prompt),
+                  return_tensors='pt').input_ids.cuda()
+
+# Generate output
+generation_output = model.generate(tokens,
+                                  streamer=streamer,
+                                  max_new_tokens=512)
+
+```
+
+### About AWQ
+
+AWQ is an efficient, accurate and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference with equivalent or better quality compared to the most commonly used GPTQ settings.
+
+AWQ models are currently supported on Linux and Windows, with NVidia GPUs only. macOS users: please use GGUF models instead.
+
+It is supported by:
+
+- [Text Generation Webui](https://github.com/oobabooga/text-generation-webui) - using Loader: AutoAWQ
+- [vLLM](https://github.com/vllm-project/vllm) - version 0.2.2 or later for support for all model types.
+- [Hugging Face Text Generation Inference (TGI)](https://github.com/huggingface/text-generation-inference)
+- [Transformers](https://huggingface.co/docs/transformers) version 4.35.0 and later, from any code or client that supports Transformers
+- [AutoAWQ](https://github.com/casper-hansen/AutoAWQ) - for use from Python code
+
+## Prompt template: ChatML
+
+```plaintext
+<|im_start|>system
+{system_message}<|im_end|>
+<|im_start|>user
+{prompt}<|im_end|>
+<|im_start|>assistant
+```
+