--- language: la license: cc-by-sa-4.0 library_name: peft base_model: google/byt5-base tags: - text2text-generation - byt5 - lora - medieval-latin - abbreviation-expansion - digital-humanities datasets: - mschonhardt/bdd-abbreviations-augmented model-index: - name: byt5-base-bdd-expansion-lora results: - task: type: text2text-generation name: Abbreviation Expansion dataset: name: mschonhardt/bdd-abbreviations-augmented type: mschonhardt/bdd-abbreviations-augmented config: default split: test metrics: - type: loss value: 0.0025 name: eval_loss --- # Model Card for byt5-base-burchard-expansion This model card describes a fine-tuned version of `google/byt5-base`, adapted for the specific task of expanding abbreviations in 11th-century Latin manuscripts from the Burchards Dekret Digital (BDD) project. - **Model type:** Byte-level sequence-to-sequence (ByT5) - **Fine-tuning method:** Low-Rank Adaptation (LoRA) with 8-bit quantization. - **Base Model:** [`google/byt5-base`](https://huggingface.co/google/byt5-base) - **Language:** Medieval Latin (`la`) - **Training Dataset:** [`mschonhardt/bdd-abbreviations-augmented`](https://huggingface.co/datasets/mschonhardt/bdd-abbreviations-augmented) - **Trainingscripts:** [Zenodo](https://doi.org/10.5281/zenodo.16628612), [Github](https://github.com/michaelscho/Abbreviationes) - **Contact:** Michael Schonhardt (michael.schonhardt@tu-darmstadt.de, [ORCID](https://orcid.org/0000-0002-2750-1900)) - **Burchards Dekret Digital (BDD):** [Website](https://www.burchards-dekret-digital.de/) - **Zenodo:** [Zenodo](https://doi.org/10.5281/zenodo.16736386) ## Model Description This repository contains the **LoRA adapters** for a ByT5-base model. It is not a standalone model but a set of trained weights that can be efficiently loaded on top of the original `google/byt5-base` to specialize it for a single task: expanding scribal abbreviations found in the manuscripts of Burchard's Decree. The ByT5 architecture was chosen because it operates directly on UTF-8 bytes, making it exceptionally robust for paleographic tasks. It requires no custom tokenizer and can handle the rich set of special Unicode characters (MUFI) and orthographic variations present in medieval texts without encountering "unknown token" issues. The model was fine-tuned using 8-bit quantization and PEFT (LoRA), which significantly reduces the computational resources required for training and inference while maintaining high performance. ## Intended Use The primary use of this model is to automate the expansion of abbreviations in texts transcribed from the five key manuscripts of the *Decretum Burchardi*. It serves as a key component in a digital editing workflow, supporting the creation of TEI-XML critical editions. ### How to Use First, install the necessary libraries. ```bash pip install transformers torch accelerate peft bitsandbytes ``` The model can be loaded with the base model (google/byt5-base) and the adapters from this repository. ```python import torch from transformers import AutoTokenizer, AutoModelForSeq2SeqLM from peft import PeftModel # Load models base_model_id = "google/byt5-base" adapter_model_id = "mschonhardt/byt5-base-bdd-expansion-lora-v4-l40s" # Load the base tokenizer and model (with 8-bit quantization) tokenizer = AutoTokenizer.from_pretrained(base_model_id) base_model = AutoModelForSeq2SeqLM.from_pretrained( base_model_id, load_in_8bit=True, device_map="auto" ) # Load the LoRA adapters onto the base model model = PeftModel.from_pretrained(base_model, adapter_model_id) model.eval() # Prepare the input text # Note the prefix used during training prefix = "expand abbreviations: " abbreviated_text = "om̅s posteri eorū cuncta sibi uendicarent sed semꝑ maiores causę sicut s̅ ep̅oꝝ..." input_text = prefix + abbreviated_text # Tokenize and generate input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to(model.device) outputs = model.generate(input_ids, max_length=1024) expanded_text = tokenizer.decode(outputs[0], skip_special_tokens=True) print(f"Abbreviated: {abbreviated_text}") print(f"Expanded: {expanded_text}") # Expected output: omnes posteri eorum cuncta sibi uendicarent sed semper maiores causę sicut sunt episcoporum... ``` ### Training and Evaluation #### Training Data The model was fine-tuned on the `mschonhardt/bdd-abbreviations-augmented dataset` (https://huggingface.co/datasets/mschonhardt/bdd-abbreviations-augmented). This dataset consists of parallel text lines extracted from the five principal manuscripts of the Decretum Burchardi. Each entry contains an abbreviated source_text and a manually verified, fully expanded target_text. Rare abbreviations were automatically multiplied to better represent them in the final model. #### Training Procedure The model was trained using the provided training script ([Zenodo](https://doi.org/10.5281/zenodo.16628612), [Github](https://github.com/michaelscho/Abbreviationes)), which uses the Hugging Face transformers and peft libraries. The model was evaluated at the end of each epoch on a held-out test split (10%) of the training data. The final model represents the checkpoint with the lowest evaluation loss of 0.0025. #### Training Hyperparameters - **learning_rate**: 3e-05 - **train_batch_size**: 32 - **eval_batch_size**: 32 - **seed**: 42 - **optimizer**: AdamW with betas=(0.9,0.999) and epsilon=1e-08 - **lr_scheduler_type**: cosine - **lr_scheduler_warmup_ratio**: 0.1 - **num_epochs**: 5 - **PEFT Method**: LoRA - **r**: 32 - **lora_alpha**: 64 - **lora_dropout**: 0.05 - **target_modules**: ["q", "k", "v", "o"] #### Framework Versions - PEFT: 0.16.0 - Transformers: 4.53.2 - Pytorch: 2.7.1+cu128 - Datasets: 4.0.0 - Tokenizers: 0.21.2 #### Limitations and Bias - **High Specificity**: This model is highly specialized. It is trained on the scribal conventions of a single scriptorium (Worms) from a specific period (early 11th century). Its performance will likely degrade significantly on manuscripts from other regions or time periods without further fine-tuning. - **Augmented Data**: As the dataset was augmented to better represent rare brevigraphs, the trained model might fail in instances where the distribution of brevigraphs differs significantly. - **Fixed Abbreviation Set**: The model can only expand abbreviations that were present in its training data. It cannot generalize to unseen brevigraphs. - **Context-Dependent**: While the 3-line window used for training provides local context, the model may still struggle with highly ambiguous abbreviations where broader semantic understanding is required. #### Citation If you use this model in your research, please cite appropriatly. ```bibtex @misc{schonhardt_byt5_burchard_2025, author = {Schonhardt, Michael}, title = {ByT5-base-burchard-expansion: A LoRA-finetuned model for Medieval Latin Abbreviation Expansion}, year = {2025}, institution = {Burchards Dekret Digital}, DOI = {https://doi.org/10.5281/zenodo.16736386}, howpublished = {\url{[https://huggingface.co/mschonhardt/byt5-base-bdd-expansion-lora-v4-l40s](https://huggingface.co/mschonhardt/byt5-base-bdd-expansion-lora-v4-l40s)}} } ```