---
library_name: transformers
tags:
  - mental-health
  - counseling
  - Mistral-7B
  - LoRA
  - quantization
  - PEFT
---
# Model Card for `mistral-7b-therapist-v1`

## Model Details

### Model Description
This is a specialized adaptation of Mistral-7B-Instruct fine-tuned for mental health counseling using parameter-efficient techniques such as 4-bit quantization (NF4) and LoRA. The model leverages the `bitsandbytes` library for quantization and the `trl` library for reinforcement learning-ready training. It has been trained on the `mental_health_counseling_conversations` dataset, formatted into chat templates with a 256-token context window.

Key features:
- **Quantization**: 4-bit quantization (NF4) reduces memory usage while maintaining performance.
- **LoRA**: Low-Rank Adaptation (r=16, alpha=32) targets attention layers for efficient fine-tuning.
- **Training**: Trained for 3 epochs with an effective batch size of 4 via gradient accumulation.
- **Metrics**: Final train loss of 0.221 and eval loss of 1.297.

### Model Sources
- **Repository**: [GitHub Repository](https://github.com/ragi19/mistral-7b-therapist-v1/import)

---

## Uses

### Direct Use
The model is designed for direct use in mental health counseling applications. It can generate empathetic, context-aware responses to user inputs, making it suitable for conversational AI systems aimed at providing psychological support.

### Downstream Use
This model can serve as a foundation for downstream tasks such as:
- Building personalized therapy chatbots.
- Enhancing existing healthcare platforms with AI-driven counseling capabilities.
- Supporting research in natural language processing for mental health.

### Out-of-Scope Use
The model is not intended for:
- Medical diagnosis or treatment without human oversight.
- High-stakes decision-making without additional validation.
- Applications requiring toxicity-free responses without safety measures.

---

## Bias, Risks, and Limitations

### Known Biases
- The model may inadvertently perpetuate biases present in the training data, particularly related to cultural or linguistic nuances.
- Responses might lack sensitivity in certain edge cases due to limited contextual understanding.

### Risks
- Misinterpretation of user input leading to inappropriate or harmful responses.
- Overreliance on the model without human intervention could lead to incorrect advice.

### Limitations
- Limited to a 256-token context window; longer conversations may lose coherence.
- Performance degradation on out-of-distribution inputs.
- Requires additional safety mechanisms (e.g., toxicity detection) for deployment.

### Recommendations
- Always validate outputs before deploying in production environments.
- Incorporate disclaimers about AI limitations and encourage users to seek professional help when necessary.
- Regularly update the model with new data to improve robustness and reduce bias.

---

## How to Get Started with the Model

Use the following code snippet to load and test the model:

```python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("ragishehab/mistral-7b-therapist-v1")
model = AutoModelForCausalLM.from_pretrained("ragishehab/mistral-7b-therapist-v1", device_map="auto")

# Generate response
input_text = "I'm feeling overwhelmed lately."
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(response)
```

---

## Training Details

### Training Data
The model was trained on the `mental_health_counseling_conversations` dataset, which contains realistic dialogues between counselors and clients. Each conversation is formatted as a sequence of user and assistant turns, ensuring alignment with conversational AI requirements.

### Training Procedure

#### Preprocessing
- Conversations were tokenized and formatted into JSON structures:
  ```json
  [{"role": "user", "content": "context"}, {"role": "assistant", "content": "response"}]
  ```
- Context windows were truncated to 256 tokens with right-padding.

#### Training Hyperparameters
- **Learning Rate**: 2e-5
- **Batch Size**: 1 (effective batch size 4 via gradient accumulation)
- **Epochs**: 3
- **Optimizer**: AdamW
- **Scheduler**: Linear decay
- **Precision**: FP16 mixed precision

#### Speeds, Sizes, Times
- Throughput: 0.139 steps/sec
- Total Steps: 1350
- GPU Memory Usage: ~3.5GB
- Training Time: ~12 hours on a single A100 GPU

---

## Evaluation

### Testing Data, Factors & Metrics

#### Testing Data
The evaluation dataset consists of unseen mental health counseling conversations, ensuring diverse scenarios and edge cases.

#### Factors
- Response relevance
- Empathy and tone appropriateness
- Coherence across multi-turn interactions

#### Metrics
- Loss: Evaluated using cross-entropy loss.
- Human Evaluation: Planned for future iterations to assess qualitative aspects like empathy and accuracy.

### Results
- Train Loss: 0.221
- Eval Loss: 1.297

#### Summary
The model demonstrates strong convergence patterns and generates coherent, context-aware responses suitable for mental health counseling.

---

## Model Examination [optional]
The model employs LoRA and quantization techniques to minimize resource consumption while maintaining performance. Future work includes interpretability studies to better understand its decision-making process.


---

## Technical Specifications

### Model Architecture and Objective
The model is based on the Mistral-7B-Instruct architecture, adapted for mental health counseling through LoRA and quantization. Its primary objective is to generate empathetic, contextually relevant responses.

### Compute Infrastructure
#### Hardware
- GPU: NVIDIA A100
- CPU: Intel Xeon Scalable Processor

#### Software
- Python: 3.9+
- PyTorch: 2.0+
- Transformers: 4.30+
- BitsAndBytes: Latest stable version

---


## Model Card Authors [optional]
- Ragishehab

---

## Model Card Contact
For inquiries or feedback, please contact ragishehab1@gmail.com.