---
base_model: meta-llama/Llama-3.2-1B
library_name: peft
license: mit
datasets:
- dair-ai/emotion
language:
- en
metrics:
- accuracy
---


# **Model Card for LLaMA-3-2-LoRA-EmotionTune**

## **Model Details**

- **Model Description:**  
  LLaMA-3-2-LoRA-EmotionTune is a causal language model fine-tuned using Low-Rank Adaptation (LoRA) on a curated emotion dataset. The dataset consists of user-generated text annotated with emotion labels (sadness, joy, love, anger, fear, or surprise). This fine-tuning enables the model to perform efficient emotion classification while preserving the core strengths of the base LLaMA-3.2-1B-Instruct model.

- **Developed by:**  
  Taha Majlesi

- **Funded by (optional):**  
  tahamajs

- **Shared by (optional):**  
  tahamajs

- **Model type:**  
  Causal Language Model with LoRA-based fine-tuning for emotion classification

- **Language(s) (NLP):**  
  English

- **License:**  
  [Choose a license, e.g., Apache-2.0, MIT]

- **Finetuned from model (optional):**  
  LLaMA-3.2-1B-Instruct

- **Model Sources (optional):**  
  Original LLaMA model and publicly available emotion datasets

- **Repository:**  
  [https://huggingface.co/your-username/LLaMA-3-2-LoRA-EmotionTune](https://huggingface.co/your-username/LLaMA-3-2-LoRA-EmotionTune)

- **Paper (optional):**  
  For LoRA: *LoRA: Low-Rank Adaptation of Large Language Models* (Hu et al., 2021)  
  For LLaMA: [Reference paper details if available]

- **Demo (optional):**  
  [Link to interactive demo if available]

---

## **Uses**

- **Direct Use:**  
  Emotion classification and sentiment analysis on short text inputs.

- **Downstream Use (optional):**  
  Can be integrated into affective computing systems, chatbots, content moderation pipelines, or any application requiring real-time sentiment detection.

- **Out-of-Scope Use:**  
  Not recommended for critical decision-making systems (e.g., mental health diagnostics) or for applications in languages other than English without further adaptation.

---

## **Bias, Risks, and Limitations**

- **Bias and Risks:**  
  The model may inherit biases present in the training data, potentially misclassifying nuanced emotions or reflecting cultural biases in emotional expression.
  
- **Limitations:**  
  - Limited to six predefined emotion categories.  
  - Performance may degrade for longer texts or in ambiguous contexts.  
  - The model is fine-tuned on a specific emotion dataset and may not generalize well across all domains.

---

## **Recommendations**

Users (both direct and downstream) should be aware of the model’s inherent biases and limitations. We recommend additional validation and fine-tuning before deploying this model in sensitive or high-stakes environments.

---

## **How to Get Started with the Model**

To load and use the model with Hugging Face Transformers and the PEFT library, try the following code snippet:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base model
base_model = AutoModelForCausalLM.from_pretrained("meta-llama/LLaMA-3.2-1B-Instruct")
tokenizer = AutoTokenizer.from_pretrained("meta-llama/LLaMA-3.2-1B-Instruct")

# Load fine-tuned LoRA adapter
model = PeftModel.from_pretrained(base_model, "your-username/LLaMA-3-2-LoRA-EmotionTune")

# Example usage:
input_text = "I feel so happy today!"
inputs = tokenizer(input_text, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

> **Demo:** An interactive demo for LLaMA-3-2-LoRA-EmotionTune is available on Hugging Face Spaces at [https://huggingface.co/spaces/your-username/demo-name](https://huggingface.co/spaces/your-username/demo-name).

---

## **Training Details**

- **Training Data:**  
  A curated emotion dataset consisting of user-generated text annotated with emotion labels (sadness, joy, love, anger, fear, surprise).

- **Training Procedure:**  
  Fine-tuning was performed on the LLaMA-3.2-1B-Instruct model using the LoRA method, which adapts selected attention layers using a low-rank approach. This method leverages a small subset of trainable parameters while keeping the majority of the model frozen.

- **Preprocessing (optional):**  
  Text normalization, tokenization using the Hugging Face tokenizer, and train/validation splitting.

- **Training Hyperparameters:**  
  - **LoRA rank (r):** 16  
  - **lora_alpha:** 32  
  - **lora_dropout:** 0.1  
  - **Learning rate:** 2e-5  
  - **Batch size:** 32  
  - **Epochs:** Early stopping applied around epoch 10 to prevent overfitting.

- **Speeds, Sizes, Times (optional):**  
  Training conducted on an NVIDIA Tesla T4 GPU for approximately 10–12 hours.

---

## **Evaluation**

- **Testing Data:**  
  A held-out subset of the emotion-annotated dataset (e.g., 100 samples).

- **Factors:**  
  The evaluation focused on the model’s ability to classify emotions within short text outputs.

- **Metrics:**  
  Accuracy and Micro F1 score.

- **Results:**  
  The fine-tuned model achieved an accuracy and Micro F1 score of approximately 31% on short-text generation tasks (5–100 token outputs), outperforming the base and instruction-tuned models.

---


## **Technical Specifications **

- **Model Architecture and Objective:**  
  Based on LLaMA-3.2-1B-Instruct, the model is fine-tuned using LoRA to specifically classify text into emotion categories.

- **Compute Infrastructure:**  
  Hugging Face Transformers, PEFT library, and PyTorch.

- **Hardware:**  
  NVIDIA Tesla T4 or equivalent GPU.

- **Software:**  
  Python, PyTorch, Hugging Face Transformers, PEFT 0.14.0.

---

## **Citation**

**BibTeX:**

```bibtex
@inproceedings{hu2021lora,
  title={LoRA: Low-Rank Adaptation of Large Language Models},
  author={Hu, Edward J and others},
  booktitle={Proceedings of ICLR},
  year={2021}
}
```

**APA:**

Hu, E. J., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. In *Proceedings of ICLR*.

---

## **Additional Information **

- **Model Card Authors:**  
  Taha Majlesi

- **Model Card Contact:**  
  tahamajlesi@ut.ac.ir

- **Framework Versions:**  
  - PEFT: 0.14.0  
  - Transformers: [version]  
  - PyTorch: [version]

---