---
library_name: transformers
tags:
- causal-lm
- vietnamese
- legal
- instruction-tuning
- lora
language: vi
license: apache-2.0
datasets:
- VTSNLP/instruct_general_dataset
- custom_legal_dataset
base_model: Qwen/Qwen2.5-3B
fine_tuned_from: Qwen/Qwen2.5-3B
model_creator: Your Name or Organization
model_type: Causal Language Model
quantization: 4-bit
inference_device: CPU
metrics:
- accuracy
new_version: Qwen/Qwen2.5-3B
pipeline_tag: question-answering
---

# Model Card Title

# Model Card for Qwen2.5-3B - John Ma

## Model Details
This model draws inspiration from John Ma, a lawyer in the TVB series Come Home Love, which I watched during my childhood. In the series, the filmmakers often included legal instructions at the end of each episode, providing valuable legal insights to viewers in Hong Kong. I found this approach both impactful and educational, sparking my motivation to create a similar resource.

This model is the result of my undergraduate thesis, designed to provide legal question-and-answer support tailored to Vietnam. It aims to enhance public understanding of legal matters, much like the series inspired greater legal awareness in its audience.

### Model Description


This model is based on the **Qwen/Qwen2.5-3B** architecture, fine-tuned using **Low-Rank Adaptation (LoRA)** for a causal language modeling task. 

The primary purpose of this model is to support legal question-and-answering tasks specific to Vietnam. It has been trained with the **VTSNLP/instruct_general_dataset** to improve its Vietnamese language capabilities, alongside a custom legal instruction dataset to enhance its understanding and response accuracy for Vietnam's legal domain. Additionally, the model is optimized with 4-bit quantization, allowing efficient deployment on cloud platforms or devices with limited hardware, without requiring a GPU.

- **Developed by:** [Do Thanh Dat - IU - HCMVNU]
- **Finetuned from model:** Qwen/Qwen2.5-3B
- **Language(s) (NLP):** Vietnamese
- **License:** [Specify license, e.g., Apache 2.0]

---

## Training Details

### Training Configuration

The LoRA configuration used during fine-tuning is as follows:

```python
config = LoraConfig(
    r=32,
    lora_alpha=32,
    lora_dropout=0.01,
    bias="none",
    task_type="CAUSAL_LM",
)
```
### Training Procedure
```python
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    packing=False,
    args=TrainingArguments(
        per_device_train_batch_size=8,
        gradient_accumulation_steps=2,
        warmup_steps=4,
        num_train_epochs=3,
        max_steps=100,
        learning_rate=2e-4,
        fp16=True,
        logging_steps=1,
        optim="adamw_8bit",
        weight_decay=0.01,
        save_steps=1000,
        lr_scheduler_type="linear",
        seed=3407,
        output_dir="qwen_v1",
        report_to="none",
    ),
)
```

### Hardware Type
NVIDIA A100 - 80GB

### Fine-Tune Method
Instruction Tuning