|
--- |
|
library_name: transformers |
|
license: other |
|
license_name: rigoclinical-nc |
|
license_link: https://huggingface.co/IIC/RigoBERTa-Clinical/blob/main/LICENSE |
|
datasets: |
|
- IIC/ClinText-SP |
|
language: |
|
- es |
|
pipeline_tag: fill-mask |
|
--- |
|
|
|
# RigoBERTa Clinical |
|
|
|
**RigoBERTa Clinical** is a state-of-the-art clinical encoder language model for Spanish, developed through domain-adaptive pretraining on the largest publicly available Spanish clinical corpus, **ClinText-SP**. This model significantly improves performance on multiple clinical NLP benchmarks while offering robust language understanding in the clinical domain. |
|
|
|
## Model Details |
|
|
|
### Model Description |
|
|
|
**RigoBERTa Clinical** was built by further pretraining the general-purpose RigoBERTa 2 on a meticulously curated clinical corpus. The pretraining leverages masked language modeling (MLM) to adapt the model’s linguistic knowledge to the Spanish clinical domain. |
|
|
|
- **Developed by:** IIC |
|
- **Model type:** Encoder |
|
- **Language(s) (NLP):** Spanish |
|
- **License:** rigoclinical-nc (permissive Non Commercial) |
|
- **Finetuned from model:** RigoBERTa 2 |
|
|
|
### Model Sources |
|
|
|
- **Paper:** [ClinText-SP and RigoBERTa Clinical: a new set of open resources for Spanish Clinical NLP](https://arxiv.org/abs/2503.18594) |
|
|
|
## Intended Use & Limitations |
|
|
|
### Intended Use |
|
|
|
**RigoBERTa Clinical** is designed for: |
|
|
|
- Clinical text understanding in Spanish. |
|
- Applications in healthcare NLP tasks such as clinical note classification, entity recognition in clinical texts, and related downstream tasks. |
|
- Research and development purposes, including benchmarking and further model adaptation. |
|
|
|
### Limitations & Caveats |
|
|
|
- **Domain Specificity:** Although highly effective for Spanish clinical texts, the model may not generalize to other domains or languages. |
|
- **Data Biases:** ClinText-SP, while the largest corpus available, may contain biases due to source selection and the inherent limitations of public clinical data. |
|
- **Operational Cost:** Despite being an encoder-based model with relatively lower computational costs compared to generative LLMs, deployment in resource-constrained settings should be carefully evaluated. |
|
|
|
## Training Details |
|
|
|
### Training Data: ClinText-SP |
|
|
|
ClinText-SP is the largest open Spanish clinical corpus and includes data from various open sources: |
|
|
|
- **Volume:** ~26 million tokens, 35,996 samples |
|
- **Sample Details:** Average of ~700 tokens per sample; contains both long-form clinical cases and shorter, schematic texts, |
|
- **Sources:** Medical journals, clinical shared tasks, radiological reports, and Wikipedia extracts. |
|
- **Availability:** [ClinText-SP](https://huggingface.co/datasets/IIC/ClinText-SP) on Hugging Face Datasets |
|
|
|
### Training Procedure |
|
|
|
#### Preprocessing |
|
|
|
- **Tokenizer:** Uses the tokenizer from RigoBERTa 2 to ensure consistency with the base model. |
|
- **Handling Long Sequences:** Clinical texts exceeding 512 tokens are segmented with a stride of 128 tokens; shorter sequences are padded as necessary. |
|
- **OOV Handling:** Out-of-vocabulary words are managed using subword tokenization, maintaining robust handling of clinical terminology. |
|
|
|
#### Training Details |
|
|
|
- **Objective:** Masked Language Modeling (MLM) |
|
- **Epochs:** 2 full epochs (with the best model selected after ~1.8 epochs, based on downstream performance) |
|
- **Hyperparameters Grid:** |
|
- **Batch Sizes:** 32, 64, 128 |
|
- **Learning Rates:** Ranges of {5e-6, 1e-5, 2e-5} for batch size 32, {1e-5, 2e-5, 4e-5} for 64, and {1e-5, 4e-5, 8e-5} for 128 |
|
- **Best Settings:** Batch size = 32, Learning rate = 2e-5, ~2800 training steps (~1.8 epochs) |
|
- **Optimizer:** AdamW with weight decay of 0.1 |
|
- **Hardware:** Trained on a single NVIDIA A100 GPU (80GB memory) |
|
|
|
## Evaluation |
|
|
|
RigoBERTa Clinical was evaluated on several Spanish clinical NLP tasks including Named Entity Recognition (NER) and multilabel classification. Evaluation metrics (F1 score and micro-averaged F1) indicate that the model outperforms previous clinical and general Spanish language models. |
|
|
|
**Key Results:** |
|
|
|
- Achieves top performance on datasets such as cantemist, meddocan, and livingner1, among others. |
|
- Consistently surpasses the performance of models that were trained solely on clinical data, demonstrating the advantage of leveraging general domain knowledge during domain adaptation. |
|
- Detailed benchmarking results and comparisons are provided in the associated publication. |
|
|
|
For a full breakdown of results (including performance on multilingual baselines and other clinical-specific models), please refer to Table 1 and the Nemenyi plot in the original paper. |
|
|
|
 |
|
|
|
## Citation |
|
|
|
If you use RigoBERTa Clinical in your research, please cite the associated paper: |
|
|
|
**BibTeX:** |
|
|
|
```bibtex |
|
@misc{subies2025clintextsprigobertaclinicalnew, |
|
title={ClinText-SP and RigoBERTa Clinical: a new set of open resources for Spanish Clinical NLP}, |
|
author={Guillem García Subies and Álvaro Barbero Jiménez and Paloma Martínez Fernández}, |
|
year={2025}, |
|
eprint={2503.18594}, |
|
archivePrefix={arXiv}, |
|
primaryClass={cs.CL}, |
|
url={https://arxiv.org/abs/2503.18594}, |
|
} |
|
``` |
|
|
|
**APA:** |
|
|
|
``` |
|
Subies, G. G., Barbero Jiménez, Á., & Martínez Fernández, P. (2025). ClinText-SP and RigoBERTa Clinical: A new set of open resources for Spanish Clinical NLP. arXiv. https://arxiv.org/abs/2503.18594 |
|
``` |
|
|
|
## Model Card Authors and Contact |
|
|
|
Guillem García Subies: [email protected], [email protected] |
|
|
