Upload folder using huggingface_hub

README.md

@@ -0,0 +1,143 @@
+---
+base_model:
+- cosimoiaia/Loquace-7B-Mistral
+license: apache-2.0
+datasets:
+- cosimoiaia/Loquace-102k
+language:
+- it
+tags:
+- bnb-my-repo
+- Italian
+- Qlora
+- Mistral
+- finetuning
+- Text Generation
+pipeline_tag: text-generation
+---
+# cosimoiaia/Loquace-7B-Mistral (Quantized)
+
+## Description
+This model is a quantized version of the original model [`cosimoiaia/Loquace-7B-Mistral`](https://huggingface.co/cosimoiaia/Loquace-7B-Mistral). 
+
+It's quantized using the BitsAndBytes library to 4-bit using the [bnb-my-repo](https://huggingface.co/spaces/bnb-community/bnb-my-repo) space.
+
+## Quantization Details
+- **Quantization Type**: int4
+- **bnb_4bit_quant_type**: nf4
+- **bnb_4bit_use_double_quant**: True
+- **bnb_4bit_compute_dtype**: float16
+- **bnb_4bit_quant_storage**: uint8
+
+
+
+# 📄 Original Model Information
+
+
+Model Card for Loquace-7B-Mistral [(Versione in Italiano tradotta da Loquace)](https://huggingface.co/cosimoiaia/Loquace-7B-Mistral/blob/main/Readme-ITA.md)
+
+# 🇮🇹 Loquace-7B-Mistral v0.1 🇮🇹 
+
+Loquace is an Italian speaking, instruction finetuned, Large Language model. 🇮🇹
+
+Loquace-7B-Mistral's peculiar features:
+
+- Is pretty good a following istructions in Italian.
+- Responds well to prompt-engineering.
+- Works well in a RAG (Retrival Augmented Generation) setup.
+- It has been trained on a relatively raw dataset [Loquace-102K](https://huggingface.co/datasets/cosimoiaia/Loquace-102k) using QLoRa and Mistral-7B-Instruct as base.
+- Training took only 4 hours on a 3090, costing a little more than <b>1 euro</b>! On [Genesis Cloud](https://gnsiscld.co/26qhlf) GPU.
+- It is <b><i>Truly Open Source</i></b>: Model, Dataset and Code to replicate the results are completely released.
+- Created in a garage in the south of Italy.
+
+The Loquace Italian LLM models are created with the goal of democratizing AI and LLM in the Italian Landscape. 
+
+<b>No more need for expensive GPU, large funding, Big Corporation or Ivory Tower Institution, just download the code and train on your dataset on your own PC (or a cheap and reliable cloud provider like [Genesis Cloud](https://gnsiscld.co/26qhlf) )</b>
+
+### Fine-tuning Instructions:
+The related code can be found at:
+https://github.com/cosimoiaia/Loquace
+
+### GGUF Version for CPU Inference:
+8bit quantized Version of Loquace can be found [here](https://huggingface.co/cosimoiaia/Loquace-7B-Mistral-GGUF)
+
+Here is an incomplate list of clients and libraries that are known to support GGUF (thanks to [TheBloke](https://huggingface.co/TheBloke) for this list and his awesome work) ):
+
+* [llama.cpp](https://github.com/ggerganov/llama.cpp). The source project for GGUF. Offers a CLI and a server option.
+* [text-generation-webui](https://github.com/oobabooga/text-generation-webui), the most widely used web UI, with many features and powerful extensions. Supports GPU acceleration.
+* [KoboldCpp](https://github.com/LostRuins/koboldcpp), a fully featured web UI, with GPU accel across all platforms and GPU architectures. Especially good for story telling.
+* [LM Studio](https://lmstudio.ai/), an easy-to-use and powerful local GUI for Windows and macOS (Silicon), with GPU acceleration.
+* [LoLLMS Web UI](https://github.com/ParisNeo/lollms-webui), a great web UI with many interesting and unique features, including a full model library for easy model selection.
+* [Faraday.dev](https://faraday.dev/), an attractive and easy to use character-based chat GUI for Windows and macOS (both Silicon and Intel), with GPU acceleration.
+* [ctransformers](https://github.com/marella/ctransformers), a Python library with GPU accel, LangChain support, and OpenAI-compatible AI server.
+* [llama-cpp-python](https://github.com/abetlen/llama-cpp-python), a Python library with GPU accel, LangChain support, and OpenAI-compatible API server.
+* [candle](https://github.com/huggingface/candle), a Rust ML framework with a focus on performance, including GPU support, and ease of use.
+
+#### Previous releases of the Loquace family:
+
+The Loquace family started in the beginning of 2023 to prove it was possible to fine-tune a Large Language model in a different language, you can find the other members of the family here:
+
+- https://huggingface.co/cosimoiaia/Loquace-70m   -   Based on pythia-70m
+- https://huggingface.co/cosimoiaia/Loquace-410m  -   Based on pythia-410m
+- https://huggingface.co/cosimoiaia/Loquace-7B    -   Based on Falcon-7B
+- https://huggingface.co/cosimoiaia/Loquace-12B   -   Based on pythia-12B
+- https://huggingface.co/cosimoiaia/Loquace-20B   -   Based on gpt-neox-20B
+
+
+These model, although not producing good results, are proof that cheap finetuning is actually possible in a short amount of time, with a relaively raw dataset and extremely low resources.
+
+## Inference:
+
+```python
+from transformers import MistralForCausalLM, AutoTokenizer
+
+
+def generate_prompt(instruction):    
+   prompt = f"""### Instruction: {instruction}
+   
+### Response:
+"""
+   return prompt
+
+model_name = "cosimoiaia/Loquace-7B-Mistral"
+
+model = MistralForCausalLM.from_pretrained(
+   model_name,
+   device_map="auto",
+   torch_dtype=torch.bfloat16                
+)
+
+model.config.use_cache = True
+
+
+tokenizer = AutoTokenizer.from_pretrained(model_name, add_eos_token=False)
+
+prompt = generate_prompt("Chi era Dante Alighieri?")
+inputs = tokenizer(prompt, return_tensors="pt").to("cuda")
+
+outputs = model.generate(**inputs, do_sample = True, num_beams = 2, top_k=50, top_p= 0.95, max_new_tokens=2046, early_stopping = True)
+print(tokenizer.decode(outputs[0], skip_special_tokens=True).split("Response:")[1].strip())
+```
+
+## Examples:
+```
+> Pretendi di essere Dante Alighieri e raccontami la tua vita.
+ Sono Dante Alighieri, un poeta e scrittore italiano del XIV secolo. Nacqui a Firenze nel 1265 e ho trascorso gran parte della mia vita viaggiando in Europa e scrivendo poesie. Ho anche scritto la Divina Commedia, una delle opere letterarie più famose di tutti i tempi.
+
+> Quanto fa 7 x 14 - 98 ?
+ 7 x 14 = 98, quindi la risposta è 0.
+
+> La radice quadrata di 144 ?
+ La radice quadrata di 144 è 12.
+
+> Inventa una ricetta pugliese con la polenta.
+ La ricetta è chiamata "Polenta al burro di arachidi e formaggio". Per prepararla, inizia bollendo la polenta secondo le istruzioni sulla confezione. Mentre la polenta si cucina, mescola insieme 1/2 tazza di burro di arachidi, 1/4 di tazza di formaggio grattugiato e 1/4 di tazza di parmigiano grattugiato. Una volta che la polenta è cotta, versala su una teglia da forno e distribuire il composto di burro di arachidi e formaggio sopra la polenta. Metti in forno a 350 gradi per 15 minuti o fino a quando la parte superiore è dorata. Servi caldo con un'insalata di pomodoro e basilico fresco.
+```
+
+## Limitations
+
+- Loquace-7B may not handle complex or nuanced queries well and may struggle with ambiguous or poorly formatted inputs.
+- The model may generate responses that are factually incorrect or nonsensical. It should be used with caution, and outputs should be carefully verified.
+
+## Model Author:
+Cosimo Iaia <[email protected]>

config.json

@@ -0,0 +1,42 @@
+{
+  "_name_or_path": "cosimoiaia/Loquace-7B-Mistral",
+  "architectures": [
+    "MistralModel"
+  ],
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 4096,
+  "initializer_range": 0.02,
+  "intermediate_size": 14336,
+  "max_position_embeddings": 32768,
+  "model_type": "mistral",
+  "num_attention_heads": 32,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 8,
+  "quantization_config": {
+    "_load_in_4bit": true,
+    "_load_in_8bit": false,
+    "bnb_4bit_compute_dtype": "float16",
+    "bnb_4bit_quant_storage": "uint8",
+    "bnb_4bit_quant_type": "nf4",
+    "bnb_4bit_use_double_quant": true,
+    "llm_int8_enable_fp32_cpu_offload": false,
+    "llm_int8_has_fp16_weight": false,
+    "llm_int8_skip_modules": null,
+    "llm_int8_threshold": 6.0,
+    "load_in_4bit": true,
+    "load_in_8bit": false,
+    "quant_method": "bitsandbytes"
+  },
+  "rms_norm_eps": 1e-05,
+  "rope_theta": 10000.0,
+  "sliding_window": 4096,
+  "tie_word_embeddings": false,
+  "torch_dtype": "float16",
+  "transformers_version": "4.49.0",
+  "use_cache": true,
+  "vocab_size": 32000
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:681d138abc599e3b6f9c533097bbd7d34e653b6bb8e4a1056b3acc4e51b90540
+size 3863534963

special_tokens_map.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,45 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": null,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [],
+  "bos_token": "<s>",
+  "chat_template": "{{ bos_token }}{% for message in messages %}{% if (message['role'] == 'user') != (loop.index0 % 2 == 0) %}{{ raise_exception('Conversation roles must alternate user/assistant/user/assistant/...') }}{% endif %}{% if message['role'] == 'user' %}{{ '[INST] ' + message['content'] + ' [/INST]' }}{% elif message['role'] == 'assistant' %}{{ message['content'] + eos_token + ' ' }}{% else %}{{ raise_exception('Only user and assistant roles are supported!') }}{% endif %}{% endfor %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "extra_special_tokens": {},
+  "legacy": true,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": null,
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": true
+}