feat: Upload fine-tuned medical NER model OpenMed-NER-OncologyDetect-ModernClinical-395M

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+openmed_vs_sota_grouped_bars.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,292 @@
+---
+widget:
+- text: "Mutations in KRAS gene drive oncogenic transformation."
+- text: "The tumor suppressor p53 pathway was disrupted."
+- text: "EGFR amplification promotes cancer cell proliferation."
+- text: "Loss of function of the PTEN gene is common in many cancers."
+- text: "The PI3K/AKT/mTOR pathway is a critical regulator of cell growth."
+tags:
+- token-classification
+- named-entity-recognition
+- biomedical-nlp
+- transformers
+- cancer-genetics
+- oncology
+- gene-regulation
+- cancer-research
+- amino_acid
+- anatomical_system
+- cancer
+- cell
+- cellular_component
+- developing_anatomical_structure
+- gene_or_gene_product
+- immaterial_anatomical_entity
+- multi-tissue_structure
+- organ
+- organism
+- organism_subdivision
+- organism_substance
+- pathological_formation
+- simple_chemical
+- tissue
+language:
+- en
+license: apache-2.0
+---
+
+# 🧬 [OpenMed-NER-OncologyDetect-ModernClinical-395M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-ModernClinical-395M)
+
+**Specialized model for Cancer Genetics - Cancer-related genetic entities**
+
+[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+[![Python](https://img.shields.io/badge/Python-3.8%2B-blue)]()
+[![Transformers](https://img.shields.io/badge/🤗-Transformers-yellow)]()
+[![OpenMed](https://img.shields.io/badge/🏥-OpenMed-green)](https://huggingface.co/OpenMed)
+
+## 📋 Model Overview
+
+This model is a **state-of-the-art** fine-tuned transformer engineered to deliver **enterprise-grade accuracy** for cancer genetics - cancer-related genetic entities. This specialized model excels at identifying and extracting biomedical entities from clinical texts, research papers, and healthcare documents, enabling applications such as **drug interaction detection**, **medication extraction from patient records**, **adverse event monitoring**, **literature mining for drug discovery**, and **biomedical knowledge graph construction** with **production-ready reliability** for clinical and research applications.
+
+### 🎯 Key Features
+- **High Precision**: Optimized for biomedical entity recognition
+- **Domain-Specific**: Trained on curated BIONLP2013_CG dataset
+- **Production-Ready**: Validated on clinical benchmarks
+- **Easy Integration**: Compatible with Hugging Face Transformers ecosystem
+
+### 🏷️ Supported Entity Types
+
+This model can identify and classify the following biomedical entities:
+
+- `B-Amino_acid`
+- `B-Anatomical_system`
+- `B-Cancer`
+- `B-Cell`
+- `B-Cellular_component`
+
+<details>
+<summary>See 27 more entity types...</summary>
+
+- `B-Developing_anatomical_structure`
+- `B-Gene_or_gene_product`
+- `B-Immaterial_anatomical_entity`
+- `B-Multi-tissue_structure`
+- `B-Organ`
+- `B-Organism`
+- `B-Organism_subdivision`
+- `B-Organism_substance`
+- `B-Pathological_formation`
+- `B-Simple_chemical`
+- `B-Tissue`
+- `I-Amino_acid`
+- `I-Anatomical_system`
+- `I-Cancer`
+- `I-Cell`
+- `I-Cellular_component`
+- `I-Developing_anatomical_structure`
+- `I-Gene_or_gene_product`
+- `I-Immaterial_anatomical_entity`
+- `I-Multi-tissue_structure`
+- `I-Organ`
+- `I-Organism`
+- `I-Organism_subdivision`
+- `I-Organism_substance`
+- `I-Pathological_formation`
+- `I-Simple_chemical`
+- `I-Tissue`
+</details>
+
+## 📊 Dataset
+
+BioNLP 2013 CG corpus targets cancer genetics entities for oncology research and cancer genomics.
+
+The BioNLP 2013 CG (Cancer Genetics) corpus is a specialized dataset focusing on cancer genetics entities and gene regulation in oncology research. This corpus contains annotations for genes, proteins, and molecular processes specifically related to cancer biology and tumor genetics. Developed for the BioNLP Shared Task 2013, it supports the development of text mining systems for cancer research, oncological studies, and precision medicine applications. The dataset is particularly valuable for identifying cancer-related biomarkers, tumor suppressor genes, oncogenes, and therapeutic targets mentioned in cancer research literature. It serves as a benchmark for evaluating NER systems used in cancer genomics, personalized medicine, and oncology informatics.
+
+
+## 📊 Performance Metrics
+
+### Current Model Performance
+- **F1 Score**: `0.85`
+- **Precision**: `0.85`
+- **Recall**: `0.85`
+- **Accuracy**: `0.94`
+
+### 🏆 Comparative Performance on BIONLP2013_CG Dataset
+
+| Rank | Model | F1 Score | Precision | Recall | Accuracy |
+|------|-------|----------|-----------|--------|-----------|
+| 🥇 1 | [OpenMed-NER-OncologyDetect-SuperMedical-355M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-SuperMedical-355M) | **0.8990** | 0.8926 | 0.9056 | 0.9416 |
+| 🥈 2 | [OpenMed-NER-OncologyDetect-ElectraMed-560M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-ElectraMed-560M) | **0.8841** | 0.8788 | 0.8895 | 0.9390 |
+| 🥉 3 | [OpenMed-NER-OncologyDetect-SnowMed-568M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-SnowMed-568M) | **0.8801** | 0.8774 | 0.8828 | 0.9366 |
+|  4 | [OpenMed-NER-OncologyDetect-PubMed-335M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-PubMed-335M) | **0.8782** | 0.8834 | 0.8730 | 0.9539 |
+|  5 | [OpenMed-NER-OncologyDetect-MultiMed-568M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-MultiMed-568M) | **0.8766** | 0.8749 | 0.8784 | 0.9351 |
+|  6 | [OpenMed-NER-OncologyDetect-SuperClinical-434M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-SuperClinical-434M) | **0.8684** | 0.8602 | 0.8768 | 0.9495 |
+|  7 | [OpenMed-NER-OncologyDetect-BioMed-335M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-BioMed-335M) | **0.8660** | 0.8540 | 0.8783 | 0.9516 |
+|  8 | [OpenMed-NER-OncologyDetect-PubMed-109M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-PubMed-109M) | **0.8606** | 0.8604 | 0.8608 | 0.9503 |
+|  9 | [OpenMed-NER-OncologyDetect-BigMed-560M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-BigMed-560M) | **0.8556** | 0.8582 | 0.8530 | 0.9250 |
+|  10 | [OpenMed-NER-OncologyDetect-ModernClinical-395M](https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-ModernClinical-395M) | **0.8471** | 0.8465 | 0.8476 | 0.9411 |
+
+
+*Rankings based on F1-score performance across all models trained on this dataset.*
+
+![OpenMed (open-source) vs. latest closed-source SOTA](https://huggingface.co/spaces/OpenMed/README/resolve/main/openmed_vs_sota_performance.png)
+
+*Figure: OpenMed (Open-Source) vs. Latest SOTA (Closed-Source) performance comparison across biomedical NER datasets.*
+
+## 🚀 Quick Start
+
+### Installation
+
+```bash
+pip install transformers torch
+```
+
+### Usage
+
+```python
+from transformers import pipeline
+
+# Load the model and tokenizer
+# Model: https://huggingface.co/OpenMed/OpenMed-NER-OncologyDetect-ModernClinical-395M
+model_name = "OpenMed/OpenMed-NER-OncologyDetect-ModernClinical-395M"
+
+# Create a pipeline
+medical_ner_pipeline = pipeline(
+    model=model_name,
+    aggregation_strategy="simple"
+)
+
+# Example usage
+text = "Mutations in KRAS gene drive oncogenic transformation."
+entities = medical_ner_pipeline(text)
+
+print(entities)
+
+token = entities[0]
+print(text[token["start"] : token["end"]])
+```
+
+NOTE: The `aggregation_strategy` parameter defines how token predictions are grouped into entities. For a detailed explanation, please refer to the [Hugging Face documentation](https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.TokenClassificationPipeline.aggregation_strategy).
+
+Here is a summary of the available strategies:
+- **`none`**: Returns raw token predictions without any aggregation.
+- **`simple`**: Groups adjacent tokens with the same entity type (e.g., `B-LOC` followed by `I-LOC`).
+- **`first`**: For word-based models, if tokens within a word have different entity tags, the tag of the first token is assigned to the entire word.
+- **`average`**: For word-based models, this strategy averages the scores of tokens within a word and applies the label with the highest resulting score.
+- **`max`**: For word-based models, the entity label from the token with the highest score within a word is assigned to the entire word.
+
+### Batch Processing
+
+For efficient processing of large datasets, use proper batching with the `batch_size` parameter:
+
+```python
+texts = [
+    "Mutations in KRAS gene drive oncogenic transformation.",
+    "The tumor suppressor p53 pathway was disrupted.",
+    "EGFR amplification promotes cancer cell proliferation.",
+    "Loss of function of the PTEN gene is common in many cancers.",
+    "The PI3K/AKT/mTOR pathway is a critical regulator of cell growth.",
+]
+
+# Efficient batch processing with optimized batch size
+# Adjust batch_size based on your GPU memory (typically 8, 16, 32, or 64)
+results = medical_ner_pipeline(texts, batch_size=8)
+
+for i, entities in enumerate(results):
+    print(f"Text {i+1} entities:")
+    for entity in entities:
+        print(f"  - {entity['word']} ({entity['entity_group']}): {entity['score']:.4f}")
+```
+
+### Large Dataset Processing
+
+For processing large datasets efficiently:
+
+```python
+from transformers.pipelines.pt_utils import KeyDataset
+from datasets import Dataset
+import pandas as pd
+
+# Load your data
+# Load a medical dataset from Hugging Face
+from datasets import load_dataset
+
+# Load a public medical dataset (using a subset for testing)
+medical_dataset = load_dataset("BI55/MedText", split="train[:100]")  # Load first 100 examples
+data = pd.DataFrame({"text": medical_dataset["Completion"]})
+dataset = Dataset.from_pandas(data)
+
+# Process with optimal batching for your hardware
+batch_size = 16  # Tune this based on your GPU memory
+results = []
+
+for out in medical_ner_pipeline(KeyDataset(dataset, "text"), batch_size=batch_size):
+    results.extend(out)
+
+print(f"Processed {len(results)} texts with batching")
+
+```
+
+### Performance Optimization
+
+**Batch Size Guidelines:**
+- **CPU**: Start with batch_size=1-4
+- **Single GPU**: Try batch_size=8-32 depending on GPU memory
+- **High-end GPU**: Can handle batch_size=64 or higher
+- **Monitor GPU utilization** to find the optimal batch size for your hardware
+
+**Memory Considerations:**
+```python
+# For limited GPU memory, use smaller batches
+medical_ner_pipeline = pipeline(
+    model=model_name,
+    aggregation_strategy="simple",
+    device=0  # Specify GPU device
+)
+
+# Process with memory-efficient batching
+for batch_start in range(0, len(texts), batch_size):
+    batch = texts[batch_start:batch_start + batch_size]
+    batch_results = medical_ner_pipeline(batch, batch_size=len(batch))
+    results.extend(batch_results)
+```
+
+## 📚 Dataset Information
+
+- **Dataset**: BIONLP2013_CG
+- **Description**: Cancer Genetics - Cancer-related genetic entities
+
+### Training Details
+- **Base Model**: BioClinical-ModernBERT-large
+- **Training Framework**: Hugging Face Transformers
+- **Optimization**: AdamW optimizer with learning rate scheduling
+- **Validation**: Cross-validation on held-out test set
+
+## 🔬 Model Architecture
+
+- **Base Architecture**: BioClinical-ModernBERT-large
+- **Task**: Token Classification (Named Entity Recognition)
+- **Labels**: Dataset-specific entity types
+- **Input**: Tokenized biomedical text
+- **Output**: BIO-tagged entity predictions
+
+## 💡 Use Cases
+
+This model is particularly useful for:
+- **Clinical Text Mining**: Extracting entities from medical records
+- **Biomedical Research**: Processing scientific literature
+- **Drug Discovery**: Identifying chemical compounds and drugs
+- **Healthcare Analytics**: Analyzing patient data and outcomes
+- **Academic Research**: Supporting biomedical NLP research
+
+## 📜 License
+
+Licensed under the Apache License 2.0. See [LICENSE](https://www.apache.org/licenses/LICENSE-2.0) for details.
+
+## 🤝 Contributing
+
+We welcome contributions of all kinds! Whether you have ideas, feature requests, or want to join our mission to advance open-source Healthcare AI, we'd love to hear from you.
+
+Follow [OpenMed Org](https://huggingface.co/OpenMed) on Hugging Face 🤗 and click "Watch" to stay updated on our latest releases and developments.
+
+

config.json

@@ -0,0 +1,115 @@
+{
+  "architectures": [
+    "ModernBertForTokenClassification"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 50281,
+  "classifier_activation": "silu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.2,
+  "classifier_pooling": "mean",
+  "cls_token_id": 50281,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "embedding_dropout": 0.0,
+  "eos_token_id": 50282,
+  "global_attn_every_n_layers": 3,
+  "global_rope_theta": 160000.0,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 1024,
+  "id2label": {
+    "0": "B-Amino_acid",
+    "1": "B-Anatomical_system",
+    "2": "B-Cancer",
+    "3": "B-Cell",
+    "4": "B-Cellular_component",
+    "5": "B-Developing_anatomical_structure",
+    "6": "B-Gene_or_gene_product",
+    "7": "B-Immaterial_anatomical_entity",
+    "8": "B-Multi-tissue_structure",
+    "9": "B-Organ",
+    "10": "B-Organism",
+    "11": "B-Organism_subdivision",
+    "12": "B-Organism_substance",
+    "13": "B-Pathological_formation",
+    "14": "B-Simple_chemical",
+    "15": "B-Tissue",
+    "16": "I-Amino_acid",
+    "17": "I-Anatomical_system",
+    "18": "I-Cancer",
+    "19": "I-Cell",
+    "20": "I-Cellular_component",
+    "21": "I-Developing_anatomical_structure",
+    "22": "I-Gene_or_gene_product",
+    "23": "I-Immaterial_anatomical_entity",
+    "24": "I-Multi-tissue_structure",
+    "25": "I-Organ",
+    "26": "I-Organism",
+    "27": "I-Organism_subdivision",
+    "28": "I-Organism_substance",
+    "29": "I-Pathological_formation",
+    "30": "I-Simple_chemical",
+    "31": "I-Tissue",
+    "32": "O"
+  },
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 2624,
+  "label2id": {
+    "B-Amino_acid": 0,
+    "B-Anatomical_system": 1,
+    "B-Cancer": 2,
+    "B-Cell": 3,
+    "B-Cellular_component": 4,
+    "B-Developing_anatomical_structure": 5,
+    "B-Gene_or_gene_product": 6,
+    "B-Immaterial_anatomical_entity": 7,
+    "B-Multi-tissue_structure": 8,
+    "B-Organ": 9,
+    "B-Organism": 10,
+    "B-Organism_subdivision": 11,
+    "B-Organism_substance": 12,
+    "B-Pathological_formation": 13,
+    "B-Simple_chemical": 14,
+    "B-Tissue": 15,
+    "I-Amino_acid": 16,
+    "I-Anatomical_system": 17,
+    "I-Cancer": 18,
+    "I-Cell": 19,
+    "I-Cellular_component": 20,
+    "I-Developing_anatomical_structure": 21,
+    "I-Gene_or_gene_product": 22,
+    "I-Immaterial_anatomical_entity": 23,
+    "I-Multi-tissue_structure": 24,
+    "I-Organ": 25,
+    "I-Organism": 26,
+    "I-Organism_subdivision": 27,
+    "I-Organism_substance": 28,
+    "I-Pathological_formation": 29,
+    "I-Simple_chemical": 30,
+    "I-Tissue": 31,
+    "O": 32
+  },
+  "layer_norm_eps": 1e-07,
+  "local_attention": 128,
+  "local_rope_theta": 10000.0,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 16,
+  "num_hidden_layers": 28,
+  "pad_token_id": 50283,
+  "position_embedding_type": "absolute",
+  "repad_logits_with_grad": false,
+  "sep_token_id": 50282,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.53.2",
+  "vocab_size": 50368
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71b192f4f6c3db5c0675f76a679caa9192ea568fa3ae2d8232363ea1ba2ff89f
+size 791748530

special_tokens_map.json

@@ -0,0 +1,37 @@
+{
+  "cls_token": {
+    "content": "[CLS]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "[MASK]",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "[PAD]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "[SEP]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

test_results.json

@@ -0,0 +1,7 @@
+{
+    "eval_accuracy": 0.9411190714634209,
+    "eval_f1": 0.8470710212545359,
+    "eval_loss": 0.8442572355270386,
+    "eval_precision": 0.8464936198058676,
+    "eval_recall": 0.8476492109430459
+}

tokenizer_config.json

@@ -0,0 +1,945 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "|||IP_ADDRESS|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "1": {
+      "content": "<|padding|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50254": {
+      "content": "                        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50255": {
+      "content": "                       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50256": {
+      "content": "                      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50257": {
+      "content": "                     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50258": {
+      "content": "                    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50259": {
+      "content": "                   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50260": {
+      "content": "                  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50261": {
+      "content": "                 ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50262": {
+      "content": "                ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50263": {
+      "content": "               ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50264": {
+      "content": "              ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50265": {
+      "content": "             ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50266": {
+      "content": "            ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50267": {
+      "content": "           ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50268": {
+      "content": "          ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50269": {
+      "content": "         ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50270": {
+      "content": "        ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50271": {
+      "content": "       ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50272": {
+      "content": "      ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50273": {
+      "content": "     ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50274": {
+      "content": "    ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50275": {
+      "content": "   ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50276": {
+      "content": "  ",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50277": {
+      "content": "|||EMAIL_ADDRESS|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50278": {
+      "content": "|||PHONE_NUMBER|||",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50279": {
+      "content": "<|endoftext|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50280": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50281": {
+      "content": "[CLS]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50282": {
+      "content": "[SEP]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50283": {
+      "content": "[PAD]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50284": {
+      "content": "[MASK]",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50285": {
+      "content": "[unused0]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50286": {
+      "content": "[unused1]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50287": {
+      "content": "[unused2]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50288": {
+      "content": "[unused3]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50289": {
+      "content": "[unused4]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50290": {
+      "content": "[unused5]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50291": {
+      "content": "[unused6]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50292": {
+      "content": "[unused7]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50293": {
+      "content": "[unused8]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50294": {
+      "content": "[unused9]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50295": {
+      "content": "[unused10]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50296": {
+      "content": "[unused11]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50297": {
+      "content": "[unused12]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50298": {
+      "content": "[unused13]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50299": {
+      "content": "[unused14]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50300": {
+      "content": "[unused15]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50301": {
+      "content": "[unused16]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50302": {
+      "content": "[unused17]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50303": {
+      "content": "[unused18]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50304": {
+      "content": "[unused19]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50305": {
+      "content": "[unused20]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50306": {
+      "content": "[unused21]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50307": {
+      "content": "[unused22]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50308": {
+      "content": "[unused23]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50309": {
+      "content": "[unused24]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50310": {
+      "content": "[unused25]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50311": {
+      "content": "[unused26]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50312": {
+      "content": "[unused27]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50313": {
+      "content": "[unused28]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50314": {
+      "content": "[unused29]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50315": {
+      "content": "[unused30]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50316": {
+      "content": "[unused31]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50317": {
+      "content": "[unused32]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50318": {
+      "content": "[unused33]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50319": {
+      "content": "[unused34]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50320": {
+      "content": "[unused35]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50321": {
+      "content": "[unused36]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50322": {
+      "content": "[unused37]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50323": {
+      "content": "[unused38]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50324": {
+      "content": "[unused39]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50325": {
+      "content": "[unused40]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50326": {
+      "content": "[unused41]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50327": {
+      "content": "[unused42]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50328": {
+      "content": "[unused43]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50329": {
+      "content": "[unused44]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50330": {
+      "content": "[unused45]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50331": {
+      "content": "[unused46]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50332": {
+      "content": "[unused47]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50333": {
+      "content": "[unused48]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50334": {
+      "content": "[unused49]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50335": {
+      "content": "[unused50]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50336": {
+      "content": "[unused51]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50337": {
+      "content": "[unused52]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50338": {
+      "content": "[unused53]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50339": {
+      "content": "[unused54]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50340": {
+      "content": "[unused55]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50341": {
+      "content": "[unused56]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50342": {
+      "content": "[unused57]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50343": {
+      "content": "[unused58]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50344": {
+      "content": "[unused59]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50345": {
+      "content": "[unused60]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50346": {
+      "content": "[unused61]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50347": {
+      "content": "[unused62]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50348": {
+      "content": "[unused63]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50349": {
+      "content": "[unused64]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50350": {
+      "content": "[unused65]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50351": {
+      "content": "[unused66]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50352": {
+      "content": "[unused67]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50353": {
+      "content": "[unused68]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50354": {
+      "content": "[unused69]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50355": {
+      "content": "[unused70]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50356": {
+      "content": "[unused71]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50357": {
+      "content": "[unused72]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50358": {
+      "content": "[unused73]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50359": {
+      "content": "[unused74]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50360": {
+      "content": "[unused75]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50361": {
+      "content": "[unused76]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50362": {
+      "content": "[unused77]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50363": {
+      "content": "[unused78]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50364": {
+      "content": "[unused79]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50365": {
+      "content": "[unused80]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50366": {
+      "content": "[unused81]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    },
+    "50367": {
+      "content": "[unused82]",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": false
+    }
+  },
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "[CLS]",
+  "extra_special_tokens": {},
+  "mask_token": "[MASK]",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "[PAD]",
+  "sep_token": "[SEP]",
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "[UNK]"
+}