Model Card for `mistral-7b-therapist-v1`

Model Details

Model Description

This is a specialized adaptation of Mistral-7B-Instruct fine-tuned for mental health counseling using parameter-efficient techniques such as 4-bit quantization (NF4) and LoRA. The model leverages the bitsandbytes library for quantization and the trl library for reinforcement learning-ready training. It has been trained on the mental_health_counseling_conversations dataset, formatted into chat templates with a 256-token context window.

Key features:

Quantization: 4-bit quantization (NF4) reduces memory usage while maintaining performance.
LoRA: Low-Rank Adaptation (r=16, alpha=32) targets attention layers for efficient fine-tuning.
Training: Trained for 3 epochs with an effective batch size of 4 via gradient accumulation.
Metrics: Final train loss of 0.221 and eval loss of 1.297.

Model Sources

Repository: GitHub Repository

Uses

Direct Use

The model is designed for direct use in mental health counseling applications. It can generate empathetic, context-aware responses to user inputs, making it suitable for conversational AI systems aimed at providing psychological support.

Downstream Use

This model can serve as a foundation for downstream tasks such as:

Building personalized therapy chatbots.
Enhancing existing healthcare platforms with AI-driven counseling capabilities.
Supporting research in natural language processing for mental health.

Out-of-Scope Use

The model is not intended for:

Medical diagnosis or treatment without human oversight.
High-stakes decision-making without additional validation.
Applications requiring toxicity-free responses without safety measures.

Bias, Risks, and Limitations

Known Biases

The model may inadvertently perpetuate biases present in the training data, particularly related to cultural or linguistic nuances.
Responses might lack sensitivity in certain edge cases due to limited contextual understanding.

Risks

Misinterpretation of user input leading to inappropriate or harmful responses.
Overreliance on the model without human intervention could lead to incorrect advice.

Limitations

Limited to a 256-token context window; longer conversations may lose coherence.
Performance degradation on out-of-distribution inputs.
Requires additional safety mechanisms (e.g., toxicity detection) for deployment.

Recommendations

Always validate outputs before deploying in production environments.
Incorporate disclaimers about AI limitations and encourage users to seek professional help when necessary.
Regularly update the model with new data to improve robustness and reduce bias.

How to Get Started with the Model

Use the following code snippet to load and test the model:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained("ragishehab/mistral-7b-therapist-v1")
model = AutoModelForCausalLM.from_pretrained("ragishehab/mistral-7b-therapist-v1", device_map="auto")

# Generate response
input_text = "I'm feeling overwhelmed lately."
inputs = tokenizer(input_text, return_tensors="pt").to("cuda")
outputs = model.generate(**inputs, max_new_tokens=100)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(response)

Training Details

Training Data

The model was trained on the mental_health_counseling_conversations dataset, which contains realistic dialogues between counselors and clients. Each conversation is formatted as a sequence of user and assistant turns, ensuring alignment with conversational AI requirements.

Training Procedure

Preprocessing

Conversations were tokenized and formatted into JSON structures:

[{"role": "user", "content": "context"}, {"role": "assistant", "content": "response"}]

Context windows were truncated to 256 tokens with right-padding.

Training Hyperparameters

Learning Rate: 2e-5
Batch Size: 1 (effective batch size 4 via gradient accumulation)
Epochs: 3
Optimizer: AdamW
Scheduler: Linear decay
Precision: FP16 mixed precision

Speeds, Sizes, Times

Throughput: 0.139 steps/sec
Total Steps: 1350
GPU Memory Usage: ~3.5GB
Training Time: ~12 hours on a single A100 GPU

Evaluation

Testing Data, Factors & Metrics

Testing Data

The evaluation dataset consists of unseen mental health counseling conversations, ensuring diverse scenarios and edge cases.

Factors

Response relevance
Empathy and tone appropriateness
Coherence across multi-turn interactions

Metrics

Loss: Evaluated using cross-entropy loss.
Human Evaluation: Planned for future iterations to assess qualitative aspects like empathy and accuracy.

Results

Train Loss: 0.221
Eval Loss: 1.297

Summary

The model demonstrates strong convergence patterns and generates coherent, context-aware responses suitable for mental health counseling.

Model Examination [optional]

The model employs LoRA and quantization techniques to minimize resource consumption while maintaining performance. Future work includes interpretability studies to better understand its decision-making process.

Technical Specifications

Model Architecture and Objective

The model is based on the Mistral-7B-Instruct architecture, adapted for mental health counseling through LoRA and quantization. Its primary objective is to generate empathetic, contextually relevant responses.

Compute Infrastructure

Hardware

GPU: NVIDIA A100
CPU: Intel Xeon Scalable Processor

Software

Python: 3.9+
PyTorch: 2.0+
Transformers: 4.30+
BitsAndBytes: Latest stable version

Model Card Authors [optional]

Ragishehab

Model Card Contact

For inquiries or feedback, please contact [email protected].

Model Card for mistral-7b-therapist-v1