Nepal Legal GPT-2 (From Scratch Implementation)

This is a custom GPT-2 style transformer model trained from scratch on Nepal's legal documents. The model was implemented entirely in PyTorch without relying on pre-trained weights, specifically designed to understand and generate text related to Nepal's legal domain.

Model Details

Model Architecture: GPT-2 style Transformer
Parameters: ~1 million
Context Length: 128 tokens
Layers: 6
Attention Heads: 6
Embedding Dimension: 384
Vocabulary Size: 50,257 (GPT-2 tokenizer)
Training Data: Nepal Legal QA English Dataset
License: MIT

Training Data Experimental

The model was trained on a specialized dataset of Nepal's legal documents in English, containing:

Legal questions and answers
Procedural instructions
Legal definitions and explanations
Court procedures and regulations