LoRA Adapter for TheBloke/Mistral-7B-Instruct-v0.2-GPTQ

This repository contains a LoRA adapter fine-tuned on the TheBloke/Mistral-7B-Instruct-v0.2-GPTQ quantized model. The adapter enables parameter-efficient fine-tuning (PEFT) without modifying the original full model weights.

Model Details

Model Description

This is a LoRA adapter trained to enhance the capabilities of the base GPTQ quantized model TheBloke/Mistral-7B-Instruct-v0.2-GPTQ, focusing on tasks such as causal language modeling and math reasoning on datasets like GSM8K.

Developed by: Dinith132
Model type: LoRA Adapter for Causal Language Model
Language(s): English
License: Apache 2.0
Finetuned from model: TheBloke/Mistral-7B-Instruct-v0.2-GPTQ

Uses

Direct Use

This LoRA adapter is intended to be loaded on top of the compatible GPTQ quantized base model for enhanced performance on tasks such as reasoning, question answering, and language generation.

Downstream Use

Users can further fine-tune this adapter or use it as a plug-in module for their specific tasks requiring low-resource fine-tuning.

Out-of-Scope Use

This adapter should not be used standalone without the compatible base model. Due to the GPTQ quantization, merging the adapter weights into the base model is not supported.

Bias, Risks, and Limitations

This model inherits biases present in the base model and training data. It may produce biased or incorrect outputs in some cases. Use with caution in sensitive applications.

Recommendations

Always validate the model outputs for your use case.
Avoid deploying in high-stakes scenarios without human oversight.
Continuously monitor for harmful or biased behavior.

How to Get Started with the Model

Installation

To use this LoRA adapter, install the required dependencies:

pip install transformers peft

Loading the Model and Tokenizer

Use the following Python code to load the base model, tokenizer, and LoRA adapter:

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel

# Load base GPTQ model
base_model = AutoModelForCausalLM.from_pretrained(
    "TheBloke/Mistral-7B-Instruct-v0.2-GPTQ",
    device_map="auto"
)

# Load tokenizer and LoRA adapter
tokenizer = AutoTokenizer.from_pretrained("Dinith132/first_lora")
model = PeftModel.from_pretrained(base_model, "Dinith132/first_lora")

Example Inference

Here’s an example of performing inference with the model:

prompt = "Alice has 20 quarters. She wants to exchange them for nickels..."

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=140)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Training Details

Training Data

The adapter was fine-tuned primarily on the GSM8K dataset, a challenging math word problem dataset.

Training Procedure

LoRA adapter fine-tuned on GPTQ quantized base model.
Used PEFT (Parameter-Efficient Fine Tuning) with LoRA configuration.

Training Hyperparameters

Hyperparameter	Value
Learning rate	2e-4
Batch size	4
Epochs	10
Optimizer	paged_adamw_8bit
Max sequence length	512

Evaluation

Evaluation was performed on subsets of the GSM8K dataset with metrics like accuracy on math reasoning problems.

Citation

If you use this adapter in your research, please cite this repository:

@misc{dinith132_lora_mistral,
  author = {Dinith132},
  title = {LoRA Adapter for TheBloke/Mistral-7B-Instruct-v0.2-GPTQ},
  year = {2025},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/Dinith132/first_lora}}
}

Model Card Authors

Dinith132

Model Card Contact

For questions, issues, or collaboration, open an issue on the Hugging Face repo or contact me directly.

Dinith132
/

first_lora