---
base_model: unsloth/llama-3.2-3b-instruct-bnb-4bit
library_name: peft
---
# Llama-3.2-3B-Instruct LoRA Fine-Tuned Model

## Overview
This is a fine-tuned LoRA (Low-Rank Adaptation) model based on the `Llama-3.2-3B-Instruct` base model. The fine-tuning process was performed for a **tipification analysis task**, targeting categories of incidents such as "ESTAFA," "ROBO," and their attempted variations.

The model leverages **LoRA** (Low-Rank Adaptation) for parameter-efficient fine-tuning, optimizing just over 24M trainable parameters while freezing the base model weights.

---

## Key Features
- **Base Model**: `unsloth/Llama-3.2-3B-Instruct`
- **Task Type**: Causal Language Modeling (CAUSAL_LM)
- **LoRA Parameters**:
  - `r`: 16
  - `lora_alpha`: 16
  - `lora_dropout`: 0.0
- **Target Modules**: `gate_proj`, `up_proj`, `down_proj`, `k_proj`, `q_proj`, `o_proj`, `v_proj`
- **Training Loss**:
  - Started at 0.779
  - Finalized at 0.6469 after 93 steps
- **Number of Trainable Parameters**: 24,313,856

---

## Dataset Distribution
The model was fine-tuned on a dataset with the following category distribution:

| **Category**             | **Count** | **Percentage** |
|--------------------------|-----------|----------------|
| ESTAFA                  | 4610      | 47.3%          |
| ROBO                    | 2307      | 23.7%          |
| HURTO                   | 2141      | 22.0%          |
| TENTATIVA DE ESTAFA     | 306       | 3.1%           |
| TENTATIVA DE ROBO       | 272       | 2.8%           |
| TENTATIVA DE HURTO      | 113       | 1.2%           |
| **Total**               | 9749      | 100%           |

---

## Training Details
- **Hardware**: Single GPU
- **Dataset Size**: ~10,000 examples
- **Epochs**: 1
- **Batch Size per Device**: 32
- **Gradient Accumulation Steps**: 4
- **Effective Total Batch Size**: 128
- **Steps**: 93

### Training Loss
| **Step** | **Training Loss** |
|----------|-------------------|
| 10       | 0.7790            |
| 20       | 0.6961            |
| 30       | 0.7048            |
| 40       | 0.6847            |
| 50       | 0.6876            |
| 60       | 0.6723            |
| 70       | 0.6443            |
| 80       | 0.6496            |
| 90       | 0.6469            |

---

## Deployment Instructions
This model is compatible with the Hugging Face Inference API and can be deployed for text generation or classification tasks. Follow the steps below to load and use the model:

### Load the Model
```python
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("<your-huggingface-username>/<repo-name>")
model = AutoModelForCausalLM.from_pretrained("<your-huggingface-username>/<repo-name>")

input_text = "TENTATIVA DE ESTAFA:"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_length=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

---

## Use Cases
This fine-tuned model is particularly useful for:
- Classifying text into predefined categories of "ESTAFA", "ROBO", "HURTO", and their "TENTATIVA DE" variations.
- Generating text based on the fine-tuned dataset categories.

---

## Fine-Tuning Colab
If you want to fine-tune a similar model, you can refer to the Colab notebook used for this fine-tuning. [Provide the Colab link if applicable]

---

## Limitations
- **Category Imbalance**: Categories such as "TENTATIVA DE HURTO" (1.2% of the dataset) and "TENTATIVA DE ROBO" (2.8% of the dataset) are underrepresented, which may affect the model’s performance on these categories.
- **Single Epoch Training**: Further training may improve performance, especially for smaller categories.

---

## Future Improvements
- **Augment dataset** for better representation of underrepresented categories.
- **Train for additional epochs** to improve classification accuracy and generalization.

---

## Contact
For questions or feedback, feel free to reach out via Hugging Face or email.

---

### Citation
If you use this model, please consider citing:
```bibtex
@misc{unsloth_lora_tipification,
  author = {Your Name},
  title = {Llama-3.2-3B-Instruct LoRA Fine-Tuned Model},
  year = {2024},
  howpublished = {\url{https://huggingface.co/<your-username>/<repo-name>}},
}
```



- PEFT 0.14.0