ShahzebKhoso/qwen2.5-instruct-0.5B-pubmedqa-lora

🧪 Qwen2.5-0.5B-Instruct + LoRA Fine-Tuned on PubMedQA (pqa_labeled)

This model is a LoRA-adapted version of Qwen2.5-0.5B-Instruct, fine-tuned using Unsloth on the pqa_labeled subset of the PubMedQA dataset.

✅ Summary

This work demonstrates that even a compact instruction-tuned model like Qwen2.5 0.5B Instruct can achieve near state-of-the-art performance on biomedical QA tasks. With LoRA fine-tuning using just 1,000 examples, this model achieves 98.99% accuracy on the PubMedQA test set.

It reframes the classification task as a text generation problem — prompting the model to generate "yes", "no", or "maybe" responses. This results in highly interpretable and efficient predictions with excellent generalization.

🔥 Key Highlights

• ✅ Model: Qwen2.5-0.5B-Instruct (general-purpose, open)
• ✅ Fine-tuning: LoRA with Unsloth
• ✅ Accuracy: 98.99%
• ✅ Macro F1: 0.977
• ✅ Very high performance on all 3 classes: yes, no, maybe
• ✅ Fully generative: no classification head
• ✅ Lightweight and deployment-friendly

📈 Evaluation Metrics

Label	Precision	Recall	F1 Score	Support
yes	0.981	1.000	0.990	52
no	1.000	1.000	1.000	38
maybe	1.000	0.889	0.941	9

• Accuracy: 98.99%
• Macro F1: 0.977
• Weighted F1: 0.989

---

🏋️ Training Configuration

• Base Model: Qwen2.5-0.5B-Instruct
• Framework: PyTorch + PEFT + Unsloth
• LoRA Config:
  • r: 16
  • alpha: 16
  • target_modules: ["q_proj", "v_proj"]
• Epochs: 100
• Batch Size: 16
• Learning Rate: 2e-4

---

💾 Usage

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

model = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-0.5B-Instruct")
model = PeftModel.from_pretrained(model, "ShahzebKhoso/qwen2.5-instruct-0.5B-pubmedqa-lora")
tokenizer = AutoTokenizer.from_pretrained("ShahzebKhoso/qwen2.5-instruct-0.5B-pubmedqa-lora")

---

⚠️ Limitations

• Trained on limited size PubMedQA subset (~1k examples)
• May still show uncertainty in "maybe" class generation
• Not suitable for medical decision-making in clinical settings

---

📚 Citation

@misc{shahzebkhoso2025qwenpubmedqa,
  title={Fine-tuning Qwen2.5-0.5B on PubMedQA with LoRA},
  author={Shahzeb Khoso},
  year={2025},
  howpublished={\\url{https://huggingface.co/ShahzebKhoso/qwen2.5-instruct-0.5B-pubmedqa-lora}},
}

---

✨ Acknowledgements

• Qwen2.5-0.5B-Instruct
• PubMedQA Dataset
• Unsloth
• Hugging Face PEFT