File size: 17,419 Bytes
291abf6 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 |
---
license: apache-2.0
language:
- ca
metrics:
- accuracy
- bleu
base_model:
- BSC-LT/salamandra-2b
---
# Aitana-2B-S
## Table of Contents
<details>
<summary>Click to expand</summary>
- [Model description](#model-description)
- [Intended uses and limitations](#intended-uses-and-limitations)
- [How to use](#how-to-use)
- [Training](#training)
- [Evaluation](#evaluation)
- [Additional information](#additional-information)
</details>
## Model description
Aitana-2B-S is a generative language model with a decoder-only architecture. This model has been trained based on Salamandra-2B, using data in Valencian to
achieve greater representation of this minority language, which is very similar to Catalan. This model has been continuously pre-trained for two epochs,
processing 2.12 billion tokens throughout the training process. Due to the data sources used, the political and administrative domains are highly present
in the model's register. The data has also been anonymised during pre-processing to avoid training with data that could violate people's privacy.
This model is based on Salamandra-2B as the basis for training and uses the same tokenizer.
## Intended uses and limitations
Aitana-2B-S is a base model that can be used for causal language modeling, it can be used as is for text generation,
although fine/instruction-tuning on specific tasks is recommended for its final use.
This language model has been trained with data in a formal register, namely related to the
administrative and political domain, so it is expected that using it in text-generation tasks
will produce text in this same format.
## How to use
```python
import torch
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
input_text = "Les corts valencianes han pres la decisió de"
model_id = "gplsi/Aitana-2B-S"
tokenizer = AutoTokenizer.from_pretrained(model_id)
generator = pipeline(
"text-generation",
model=model_id,
tokenizer=tokenizer,
torch_dtype=torch.bfloat16,
trust_remote_code=True,
device_map="auto",
)
generation = generator(
input_text,
do_sample=True,
top_k=10,
eos_token_id=tokenizer.eos_token_id,
)
print(f"Result: {generation[0]['generated_text']}")
```
## Training
### Training data
The training corpus has been obtained using web scraping on public data from different sources such as the
[Official Gazette of the University of Alicante (BOUA)](https://www.boua.ua.es/ca), [the Official Gazette of the Generalitat Valenciana (DOGV)](https://dogv.gva.es/va) and accurate data provided by
[the Valencian Courts (DSCV and DSCCV)](https://www.cortsvalencianes.es/ca-va/). Giving a total of 1.304 million tokens, according to the following table.
Dataset | Language | Total Sentences | Total Words | Total Numbers | Other Symbols | Unique Words | Total Tokens | Average sentence Length | Average Word Length |
|---------|----------|--------------------|-------------|---------------|---------------|--------------|--------------|-------------------------|---------------------|
**BOUA** | va | 0.606M | 12.355M | 0.488M | 0.055M | 0.211M | 12.899M | 21.27 | 4.89
**DOGCV** | va | 4.569M | 50.566M | 6.339M | 0.613M | 17.436M | 57.517M | 12.59 | 4.68
**DOGV** | va | 18.598M | 311.380M | 24.138M | 2.731M | 11.416M | 338.250M | 18.19 | 4.88
**DSCCV** | va | 2.353M | 46.116M | 0.554M | 2.352m | 5.031M | 46.672M | 19.84 | 4.56
**DSCV** | va | 1.646M | 32.496M | 0.433M | 1.427m | 3.796M | 32.930M | 20.01 | 4.65
**UN** | va | 0.394M | 12.289M | 0.253M | 0.015M | 0.533M | 12.556M | 31.86 | 4.86
**VJ** | va | 0.913M | 23.594M | 0.466M | 23.314m | 0.849M | 24.084M | 26.39 | 4.57
Several of the downloaded sources have already been used in the Meta-Llama-3-8B training, so the date of data collection for the previous
model has been taken into account and those web pages have been scraped from that date.
Information on the datasets used for training is shown below:
- Official Bulletin of the University of Alicante (BOUA): These are the documents issued by the University of Alicante related to grants, regulations, and different resolutions of laws published periodically, specifically the Valencian version.
- Legacy Official Journal of the Generalitat Valenciana (DOGCV): This journal contains historical documents issued by the Valencian Community. These documents were initially recorded on paper and digitised with the standardisation of the digital format. They have the same subject matter as the DOGV documents but were generated between 1980 and 1997.
- Official Journal of the Generalitat Valenciana (DOGV): These documents contain official communications of the Valencian Community. They mainly deal with issuing laws, legal measures, and public sector communication. These journals were issued from 1998 to 2023.
- Valencian Parliament Diary Dataset (DSCCV) contains records from various committee meetings held in the parliament, with each meeting documented in a separate text file.
- Journal of the Valencian Parliament (DSCV): in this case, the transcripts of the different meetings held in the parliament's plenary sessions, with data from 1999 to 2022.
- University news (UN): We have news in a colloquial register from different universities that have Valencian as an official language, including the universities of Valencia, Alicante, Jaume I, and the Polytechnic University of Valencia.
- Valencian Journals (VJ): These include various types of Valencian journals with colloquial records to facilitate daily record-keeping alongside the legal and bureaucratic documents from previous records. These include a total of 10 different journals.
### Training parameters
During the training of the model, a high context window was desired when generating text, so it was decided to use an input size of 2048
tokens and a minimum context window of 512 in case of truncating the input sequences. 80% of the data obtained was used for the training stage,
while 20% was used during the evaluation stage. A summary of the parameters used during training can be seen in the following table:
Parameter | Value |
|-------------------------------|-------|
Epochs | 2 |
Learning Rate | 2e-5 |
Warmup Steps | 0 |
Precision | bf-16 |
Weight decay | 1e-1 |
Training Fraction | 0.95 |
Evaluation Fraction | 0.05 |
Input size (tokens) | 2048 |
Minimum context window (tokens) | 512 |
### Distributed Training Strategy
A distributed training strategy called Fully Sharded Data Parallel (FSDP)
has been used. With this, the entire model has been loaded among the 4 A100s available for training with a mini-batch size of size 1 and a total gradient accumulation step of 64.
### Languages
In addition to the data already used for the training of Meta-Llama-3-8B, data completely in Valencian from the sources mentioned in
the previous section has been used.
## Evaluation
In the following table, we can see the results obtained with different benchmarks in comparison with
the model used for continuous pre-training. The results have been obtained from the model pre-trained;
no instruction tuning or fine-tuning of any kind has been performed.
### Valencian
#### Classification Benchmarks
| Dataset | Lang. | Task | Metric | Salamandra-2B | Aitana-2B-S |
|------------------------------|--------|----------------------------|-------------|-----------|-----------------------|
| XNLI | va |Natural Language Inference | acc | **0.475** | 0.473 |
#### Generation Benchmarks
| Dataset | Lang. | Task | Metric | Salamandra-2B | Aitana-2B-S |
|------------------------------|--------|----------------------------|-------------|---------------|-----------------------|
| Cocoteros | va |Reading Comprehension | bleu | **6.32** | 5.76 |
| Phrases ca-va | va-ca |Translation - Adaptation | bleu | 79.82 | **81.92** |
| Phrases va-ca | va-ca |Translation - Adaptation | bleu | **78.05** | 76.53 |
| Phrases va-es | va-es |Translation | bleu | **76.04** | 75.99 |
| Phrases es-va | es-va |Translation | bleu | 58.86 | **61.51** |
### Catalan
#### Classification Benchmarks
| Dataset | Lang. | Task | Metric | Salamandra-2B | Aitana-2B-S |
|------------------------------|--------|---------------------------|-------------|---------------|-------------------|
| Belebele Cat_latn | ca | Reading Comprehension | acc | 0.231 | **0.257** |
| COPA | ca | Commonsense Reasoning | acc | 0.700 | **0.712** |
| XStoryCloze | ca | Commonsense Reasoning | acc | 0.655 | **0.657** |
| OpenBookQA | ca | Question Answering | acc | **0.294** | 0.282 |
| PAWS | ca | Paraphrasing | acc | **0.556** | 0.551 |
| PiQA | ca | Question Answering | acc | 0.643 | **0.646** |
| SiQA | ca | Question Answering | acc | **0.434** | 0.432 |
| ARC Easy | ca | Question Answering | acc | **0.551** | 0.549 |
| ARC Challenge | ca | Question Answering | acc | **0.290** | 0.288 |
| XNLI | ca | Natural Language Inference| acc | 0.473 | **0.480** |
| Teca | ca | Natural Language Inference| acc | **0.465** | 0.459 |
| WNLI | ca | Natural Language Inference| acc | **0.577** | 0.563 |
| Catcola | ca | Linguistic Acceptability | acc | **0.543** | 0.525 |
| Catcola | ca | Linguistic Acceptability | mcc | **0.046** | 0.023 |
| Catalanqa | ca | Question Answering | F1 | **0.668** | 0.655 |
| Mgsm direct | ca | Math | exact match | 0.024 | **0.028** |
| Catalanqa | ca | Question Answering | exact match | **0.437** | 0.415 |
| Xquad | ca | Question Answering | exact match | **0.371** | 0.354 |
| Xquad | ca | Question Answering | F1 | **0.579** | 0.566 |
#### Generation Benchmarks
| Dataset | Lang. | Task | Metric | Salamandra-2B | Aitana-2B-S |
|------------------------------|--------|--------------------------|--------|----------------|-----------------------|
| Cabreu abstractive | ca | Summarization | bleu | 5.78 | **6.24** |
| Cabreu extractive | ca | Summarization | bleu | **42.89** | 41.19 |
| Cabreu extreme | ca | Summarization | bleu | 3.29 | **3.81** |
### Spanish
#### Classification Benchmarks
| Dataset | Lang. | Task | Metric | Salamandra-2B | Aitana-2B-S |
|------------------------------|--------|---------------------------|-------------|-----------|-----------------------|
| Belebele Cat_latn | es | Reading Comprehension | acc | **0.228** | 0.224 |
| PAWS | es | Paraphrasing | acc | **0.561** | 0.543 |
| XNLI | es | Natural Language Inference| acc | **0.439** | 0.422 |
| WNLI | es | Natural Language Inference| acc | 0.563 | 0.563 |
| XStoryCloze | es | Commonsense Reasoning | acc | **0.653** | 0.652 |
| Escola | es | Linguistic Acceptability | acc | **0.593** | 0.536 |
| Escola | es | Linguistic Acceptability | mcc | **0.031** | 0.010 |
| OpenbookQA | es | Question Answering | acc | 0.308 | **0.314** |
| MGSM Direct | es | Math | exact match | 0.020 | 0.020 |
| XQUAD | es | Question Answering | exact match | **0.377** | 0.373 |
| XQUAD | es | Question Answering | F1 | **0.584** | 0.583 |
#### Generation Benchmarks
| Dataset | Lang. | Task | Metric | Salamandra-2B | Aitana-2B-S |
|------------------------------|--------|----------------|---------|-------------|-----------------------|
| Cocoteros | es |Reading Comprehension| bleu | **8.46** | 7.35 |
| XLSum | es | Summarization | bleu | **0.801** | 0.434 |
### English
#### Classification Benchmarks
| Dataset | Lang. | Task | Metric | Salamandra-2B | Aitana-2B-S |
|------------------------------|--------|----------------------------|-------------|-----------|-----------------------|
| Arc Challenge | en | Question Answering | acc | 0.370 | **0.374** |
| Arc Easy | en | Question Answering | acc | **0.722** | 0.719 |
| Belebele Eng_latn | en | Reading Comprehension | acc | 0.216 | **0.229** |
| PAWS | en | Paraphrasing | acc | 0.561 | **0.562** |
| XNLI | en | Natural Language Inference | acc | **0.462** | 0.446 |
| XStoryCloze | en | Commonsense Reasoning | acc | 0.711 | **0.713** |
| OpenBookQA | en | Question Answering | acc | 0.300 | **0.308** |
| PiQA | en | Question Answering | acc | 0.737 | **0.743** |
| Social iqa | en | Question Answering | acc | **0.454** | 0.451 |
| WNLI | en | Natural Language Inference | acc | 0.465 | **0.578** |
| MGSM Direct | en | Math | exact match | 0.064 | 0.064 |
| TriviaQA | en | Question Answering | exact match | -0.019 | **0.015** |
## Additional information
### Author
Language and Information System Group GPLSI
### Contact
For further information, please send an email to GPLSI
### Copyright
Copyright(c) 2025 by GPLSI(https://gplsi.dlsi.ua.es/).
### License
[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0)
### Funding
This work was funded by [ILENIA](https://proyectoilenia.es/)-[VIVES](https://vives.gplsi.es/) project <<2022/TL22/00215334>>
### Disclaimer
The model published in this repository is intended for a generalist purpose and is available to third parties under a permissive Apache License, Version 2.0.
Be aware that the model may have biases and/or any other undesirable distortions.
When third parties deploy or provide systems and/or services to other parties using this model (or any system based on it) or become users of the model, they should note that it is their responsibility to mitigate the risks arising from its use and, in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.
In no event shall the owner and creator of the model (GPLSI) be liable for any results arising from the use made by third parties. |