File size: 5,507 Bytes
80f8d4f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 |
---
library_name: transformers
license: other
license_name: rigoclinical-nc
license_link: https://huggingface.co/IIC/RigoBERTa-Clinical/blob/main/LICENSE
datasets:
- IIC/ClinText-SP
language:
- es
pipeline_tag: fill-mask
---
# RigoBERTa Clinical
**RigoBERTa Clinical** is a state-of-the-art clinical encoder language model for Spanish, developed through domain-adaptive pretraining on the largest publicly available Spanish clinical corpus, **ClinText-SP**. This model significantly improves performance on multiple clinical NLP benchmarks while offering robust language understanding in the clinical domain.
## Model Details
### Model Description
**RigoBERTa Clinical** was built by further pretraining the general-purpose RigoBERTa 2 on a meticulously curated clinical corpus. The pretraining leverages masked language modeling (MLM) to adapt the model’s linguistic knowledge to the Spanish clinical domain.
- **Developed by:** IIC
- **Model type:** Encoder
- **Language(s) (NLP):** Spanish
- **License:** rigoclinical-nc (permissive Non Commercial)
- **Finetuned from model:** RigoBERTa 2
### Model Sources
- **Paper:** [ClinText-SP and RigoBERTa Clinical: a new set of open resources for Spanish Clinical NLP](https://arxiv.org/abs/2503.18594)
## Intended Use & Limitations
### Intended Use
**RigoBERTa Clinical** is designed for:
- Clinical text understanding in Spanish.
- Applications in healthcare NLP tasks such as clinical note classification, entity recognition in clinical texts, and related downstream tasks.
- Research and development purposes, including benchmarking and further model adaptation.
### Limitations & Caveats
- **Domain Specificity:** Although highly effective for Spanish clinical texts, the model may not generalize to other domains or languages.
- **Data Biases:** ClinText-SP, while the largest corpus available, may contain biases due to source selection and the inherent limitations of public clinical data.
- **Operational Cost:** Despite being an encoder-based model with relatively lower computational costs compared to generative LLMs, deployment in resource-constrained settings should be carefully evaluated.
## Training Details
### Training Data: ClinText-SP
ClinText-SP is the largest open Spanish clinical corpus and includes data from various open sources:
- **Volume:** ~26 million tokens, 35,996 samples
- **Sample Details:** Average of ~700 tokens per sample; contains both long-form clinical cases and shorter, schematic texts,
- **Sources:** Medical journals, clinical shared tasks, radiological reports, and Wikipedia extracts.
- **Availability:** [ClinText-SP](https://huggingface.co/datasets/IIC/ClinText-SP) on Hugging Face Datasets
### Training Procedure
#### Preprocessing
- **Tokenizer:** Uses the tokenizer from RigoBERTa 2 to ensure consistency with the base model.
- **Handling Long Sequences:** Clinical texts exceeding 512 tokens are segmented with a stride of 128 tokens; shorter sequences are padded as necessary.
- **OOV Handling:** Out-of-vocabulary words are managed using subword tokenization, maintaining robust handling of clinical terminology.
#### Training Details
- **Objective:** Masked Language Modeling (MLM)
- **Epochs:** 2 full epochs (with the best model selected after ~1.8 epochs, based on downstream performance)
- **Hyperparameters Grid:**
- **Batch Sizes:** 32, 64, 128
- **Learning Rates:** Ranges of {5e-6, 1e-5, 2e-5} for batch size 32, {1e-5, 2e-5, 4e-5} for 64, and {1e-5, 4e-5, 8e-5} for 128
- **Best Settings:** Batch size = 32, Learning rate = 2e-5, ~2800 training steps (~1.8 epochs)
- **Optimizer:** AdamW with weight decay of 0.1
- **Hardware:** Trained on a single NVIDIA A100 GPU (80GB memory)
## Evaluation
RigoBERTa Clinical was evaluated on several Spanish clinical NLP tasks including Named Entity Recognition (NER) and multilabel classification. Evaluation metrics (F1 score and micro-averaged F1) indicate that the model outperforms previous clinical and general Spanish language models.
**Key Results:**
- Achieves top performance on datasets such as cantemist, meddocan, and livingner1, among others.
- Consistently surpasses the performance of models that were trained solely on clinical data, demonstrating the advantage of leveraging general domain knowledge during domain adaptation.
- Detailed benchmarking results and comparisons are provided in the associated publication.
For a full breakdown of results (including performance on multilingual baselines and other clinical-specific models), please refer to Table 1 and the Nemenyi plot in the original paper.
## Citation
If you use RigoBERTa Clinical in your research, please cite the associated paper:
**BibTeX:**
```bibtex
@misc{subies2025clintextsprigobertaclinicalnew,
title={ClinText-SP and RigoBERTa Clinical: a new set of open resources for Spanish Clinical NLP},
author={Guillem García Subies and Álvaro Barbero Jiménez and Paloma Martínez Fernández},
year={2025},
eprint={2503.18594},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2503.18594},
}
```
**APA:**
```
Subies, G. G., Barbero Jiménez, Á., & Martínez Fernández, P. (2025). ClinText-SP and RigoBERTa Clinical: A new set of open resources for Spanish Clinical NLP. arXiv. https://arxiv.org/abs/2503.18594
```
## Model Card Authors and Contact
Guillem García Subies: [email protected], [email protected]
|