meta-llama/Llama-3.2-3B Fine-tuned with QLora
This model is a fine-tuned version of meta-llama/Llama-3.2-3B using the LoRA on the google/boolq dataset.
🚀 Training Details
Fine-tuning Configuration
- Base Model: meta-llama/Llama-3.2-3B
- Quantization: 4-bit compute.
- LoRA Rank: 16
- LoRA Alpha: 32
- Batch Size: 8 (per device)
- Gradient Accumulation: 4
- Learning Rate: 2e-5
- Sequence Length: 1024 tokens
- Gradient Checkpointing: Enabled
📊 Training Metrics
- Total Steps: 295
- Final Loss: 1.618368478548729
- Trainable Params: 24,313,856
## ⚖️ License
This model inherits the Apache 2.0 license.
- Downloads last month
- 5
Model tree for D1zzYzz/BOOLQ-QLORA-llama-3.2-3B
Base model
meta-llama/Llama-3.2-3B