---
tags:
- llama
- alpaca
- qlora
- unsloth
- instruction-tuning
- fine-tuned
base_model: unsloth/llama-3-3b-bnb-4bit
library_name: peft
license: apache-2.0
datasets:
- tatsu-lab/alpaca
language:
- en
pipeline_tag: text-generation
---

# LLaMA-3 3B Fine-tuned with QLoRA (Unsloth) on Alpaca

This model is a fine-tuned version of [`unsloth/llama-3-3b-bnb-4bit`](https://huggingface.co/unsloth/llama-3-3b-bnb-4bit) using **QLoRA** and [Unsloth](https://github.com/unslothai/unsloth) for efficient instruction-tuning.

## 📖 Training Details
- **Dataset**: [`tatsu-lab/alpaca`](https://huggingface.co/datasets/tatsu-lab/alpaca)
- **QLoRA**: 4-bit quantization (NF4) using `bitsandbytes`
- **LoRA Rank**: 4 (adjust based on your config)
- **LoRA Alpha**: 8
- **Batch Size**: 2 per device
- **Gradient Accumulation**: 4
- **Learning Rate**: 2e-4
- **Epochs**: 1
- **Trainer**: `trl.SFTTrainer`

## 💡 Notes
- Optimized for memory-efficient fine-tuning with Unsloth
- LoRA adapters are injected into Q, O, V attention projections
- No evaluation was run during training — please evaluate separately

## 📝 License
Apache 2.0