Mistral-7B Fine-Tuned for Academic Style (QLoRA)

This is a parameter-efficient fine-tuning of mistralai/Mistral-7B-v0.1 using QLoRA on 500K academic abstracts. It was built for the ECS 271 final project at UC Davis.

Intended Use

The model is designed to generate formal academic paragraphs given a paper title, useful for research drafts, educational AI tools, and academic-style assistants.

Training Details

Base model: mistralai/Mistral-7B-v0.1
Method: QLoRA (low-rank adapter)
Prompt format: "Write an academic paragraph given the title: ..."
Dataset: 500K arXiv abstracts
Epochs: 1
GPU: RTX 5070 Ti (~60 hours)

Limitations

Generic or templated outputs
No citation support
Frequent hallucinations

Example

Prompt:
"Write an academic paragraph given the title: LoRA for In-Context Learning"

Output:
"We present LoRA (Loosely Regularized Adapters), a novel approach to fine-tune large language models for in-context learning tasks. Unlike traditional methods, LoRA updates only a small number of trainable parameters, achieving comparable performance while reducing training costs."

More Info

Main Project: Github