AshishKJain/gpt2-large-alpaca-sft

GPT2-Large Fine-tuned on Alpaca Instructions

Model Description

This is a GPT2-Large (774M parameters) model fine-tuned on the Stanford Alpaca instruction dataset. The model has been trained to follow instructions and generate appropriate responses in an instruction-following format.

Model Details

• Base Model: GPT2-Large (774M parameters)
• Training Data: Stanford Alpaca dataset (52,002 instruction-response pairs)
• Fine-tuning Method: Supervised Fine-tuning (SFT)
• Model Architecture:
  • Layers: 36
  • Hidden Size: 1280
  • Attention Heads: 20
  • Context Length: 1024
  • Vocabulary Size: 50257

Training Details

• Training Split: 85% training, 10% test, 5% validation
• Prompt Format: Enhanced Alpaca format
• Optimizer: AdamW
• Mixed Precision: BF16
• Context Window: 1024 tokens
• Accuracy on Alpaca test set: 30-40

Usage

from transformers import GPT2LMHeadModel, GPT2Tokenizer

# Load model and tokenizer
model = GPT2LMHeadModel.from_pretrained("AshishKJain/gpt2-large-alpaca-sft")
tokenizer = GPT2Tokenizer.from_pretrained("AshishKJain/gpt2-large-alpaca-sft")

# Prepare prompt in Alpaca format
prompt = """Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
Give three tips for staying healthy.
### Response:
"""

# Generate response
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(
    inputs['input_ids'],
    max_new_tokens=256,
    temperature=0.7,
    top_k=50,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

Prompt Format

The model expects prompts in the following format:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
[Your instruction here]
### Response:

For tasks with additional input context:

Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
[Your instruction here]
### Input:
[Additional context]
### Response:

Limitations and Biases

• This model inherits biases from both GPT2-Large and the Alpaca dataset
• The Alpaca dataset was generated using GPT-3.5, which may have limitations
• The model may generate incorrect or biased information
• Context window is limited to 1024 tokens

License

This model is released under the MIT License. However, note that:

• GPT2 is released under the Modified MIT License
• Alpaca dataset has specific usage guidelines from Stanford

Citation

If you use this model, please cite:

@misc{gpt2-large-alpaca,
  author = {Ashish K Jain},
  title = {GPT2-Large Fine-tuned on Alpaca Instructions},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/AshishKJain/gpt2-large-alpaca-sft}}
}

Acknowledgments

• OpenAI for GPT2
• Stanford for the Alpaca dataset
• HuggingFace for the transformers library