Qwen2.5-3B-Instruct-Books-19K

This model is a fine-tuned version of Qwen/Qwen2.5-3B-Instruct on the Books dataset for Essay Grading.

• Workflow: GitHub Repository: https://github.com/IsmaelMousa/automatic-essay-grading.
• Base Model: Qwen2.5-3B-Instruct: https://doi.org/10.48550/arXiv.2412.15115.
• Fine-tuning Dataset: Books-19K: https://github.com/IsmaelMousa/Books/19K.
• Task: Automatic Essay Grading (Text Generation).

Dataset

The Books dataset is a synthetic collection of essay-style data points generated using public domain literature and large language model prompting. The dataset comprises a total of 300 entries and is built from six classic books. Four of these: The Life of James Watt, The Life of Julius Caesar, The Moonstone, and North and South; were used during the training phase, while the remaining two: The Life of Napoleon and Sense and Sensibility; were held out for benchmarking purposes. Each book contributed exactly 50 entries, leading to a structured split of 200 training samples and 100 benchmark samples.

All entries were generated using Le Chat Mistral, a model developed by Mistral AI. A carefully crafted prompt was used to ensure each generated entry included a question, a reference answer written by an expert, a student answer meant to simulate a real-world response, a mark scheme outlining the grading criteria, a score between 1 and 4, and a rationale explaining why the score was assigned. The prompt enforced strict quality control: no duplicate questions or answers were allowed, all required fields had to be present, and the scoring range was strictly limited to valid values. The final output was formatted as CSV files to maintain consistency and ensure compatibility with downstream processing.

For more details, the metadata can be accessed at: metadata.

Modeling

The modeling approach for this study was carefully designed to evaluate the performance of different large language models (LLMs) on the automated essay grading task. We selected the Qwen2.5 architecture to represent a range of model sizes: 0.5B, 1.5B, and 3B. Each model was instruction-tuned on the Books dataset in varying sizes, with hyperparameters optimized to balance computational efficiency and performance. The experiments were conducted on GPU-accelerated hardware, leveraging techniques such as gradient checkpointing, flash attention, and mixed-precision training to maximize resource utilization.

Evaluation

The evaluation methodology employed both quantitative metrics and qualitative analysis. For quantitative assessment, we computed accuracy, precision, recall, F1 score, root mean squared error (RMSE), and Cohen's kappa score (CKS) for the scoring task, while using BERT-Score precision, recall, and F1 for rationale evaluation. On a held-out test set of 100 samples. Qualitative examination of models' outputs revealed cases where most of the models correctly identified key aspects of student answers but sometimes failed to properly align its scoring with the rubric criteria.

Evaluation results for score and rationale outputs:

Aspect	F1	Precision	Recall	Accuracy	CKS	RMSE
Score	0.1732	0.1889	0.2492	0.2200	-0.0005	1.7944
Rationale	0.5449	0.5567	0.5371	--	--	--

Usage

Below is an example of how to use the model with the Hugging Face Transformers library:

from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline
import torch


checkpoint       = "IsmaelMousa/Qwen2.5-3B-Instruct-Books-19K"
device           = torch.device("cuda" if torch.cuda.is_available() else "cpu")

tokenizer        = AutoTokenizer       .from_pretrained(checkpoint)
model            = AutoModelForCausalLM.from_pretrained(checkpoint)

assistant        = pipeline("text-generation", tokenizer=tokenizer, model=model, device=device)

question         = input("Question        : ")
reference_answer = input("Reference Answer: ")
student_answer   = input("Student Answer  : ")
mark_scheme      = input("Mark Scheme     : ")

system_content   = "You are a grading assistant. Evaluate student answers based on the mark scheme. Respond only in JSON format with keys 'score' (int) and 'rationale' (string)."

user_content     = ("Provide both a score and a rationale by evaluating the student's answer strictly within the mark scheme range,"
                    " grading based on how well it meets the question's requirements by comparing the student answer to the reference answer.\n"
                    f"Question: {question}\n"
                    f"Reference Answer: {reference_answer}\n"
                    f"Student Answer: {student_answer}\n"
                    f"Mark Scheme: {mark_scheme}")

messages         = [{"role": "system", "content": system_content}, {"role": "user", "content": user_content}]

inputs           = tokenizer.apply_chat_template(messages, tokenize=False)

output           = assistant(inputs, max_new_tokens=128, do_sample=False, return_full_text=False)[0]["generated_text"]

print(output)

Frameworks

• datasets-3.6.0
• torch-2.7.0
• transformers-4.51.3
• trl-0.17.0
• scikit-learn-1.6.1
• bert-score-0.3.13
• json-repair-0.46.0