Upload 8 files

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -1,3 +1,300 @@
----
-license: apache-2.0
----
+---
+license: apache-2.0
+language:
+- ca
+metrics:
+- accuracy
+- bleu
+base_model:
+- BSC-LT/salamandra-2b
+---
+# Aitana-2B-S
+
+## Table of Contents
+<details>
+<summary>Click to expand</summary>
+
+- [Model description](#model-description)
+- [Intended uses and limitations](#intended-uses-and-limitations)
+- [How to use](#how-to-use)
+- [Training](#training)
+- [Evaluation](#evaluation)
+- [Additional information](#additional-information)
+
+
+</details>
+
+
+## Model description
+
+Aitana-2B-S is a generative language model with a decoder-only architecture. This model has been trained based on Salamandra-2B, using data in Valencian to 
+achieve greater representation of this minority language, which is very similar to Catalan. This model has been continuously pre-trained for two epochs, 
+processing 2.12 billion tokens throughout the training process. Due to the data sources used, the political and administrative domains are highly present 
+in the model's register. The data has also been anonymised during pre-processing to avoid training with data that could violate people's privacy.
+
+This model is based on Salamandra-2B as the basis for training and uses the same tokenizer.
+
+
+## Intended uses and limitations
+
+Aitana-2B-S is a base model that can be used for causal language modeling, it can be used as is for text generation,
+although fine/instruction-tuning on specific tasks is recommended for its final use.
+
+This language model has been trained with data in a formal register, namely related to the
+administrative and political domain, so it is expected that using it in text-generation tasks
+will produce text in this same format.
+
+
+## How to use
+```python
+import torch
+from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM
+input_text = "Les corts valencianes han pres la decisió de"
+model_id  = "gplsi/Aitana-2B-S"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+generator = pipeline(
+    "text-generation",
+    model=model_id,
+    tokenizer=tokenizer,
+    torch_dtype=torch.bfloat16,
+    trust_remote_code=True,
+    device_map="auto",
+)
+generation = generator(
+    input_text,
+    do_sample=True,
+    top_k=10,
+    eos_token_id=tokenizer.eos_token_id,
+)
+print(f"Result: {generation[0]['generated_text']}")
+
+```
+
+## Training
+
+### Training data
+
+The training corpus has been obtained using web scraping on public data from different sources such as the 
+[Official Gazette of the University of Alicante (BOUA)](https://www.boua.ua.es/ca), [the Official Gazette of the Generalitat Valenciana (DOGV)](https://dogv.gva.es/va) and accurate data provided by 
+[the Valencian Courts (DSCV and DSCCV)](https://www.cortsvalencianes.es/ca-va/). Giving a total of 1.304 million tokens, according to the following table.
+
+
+Dataset	  | Language | Total Sentences    | Total Words | Total Numbers | Other Symbols | Unique Words | Total Tokens | Average sentence Length | Average Word Length |
+|---------|----------|--------------------|-------------|---------------|---------------|--------------|--------------|-------------------------|---------------------|
+**BOUA**  | va	     | 0.606M	          |   12.355M   |    0.488M     |    0.055M     |   0.211M     |   12.899M    |         21.27           |      4.89
+**DOGCV** | va	     | 4.569M	          |   50.566M   |    6.339M     |    0.613M     |   17.436M    |   57.517M    |         12.59           |      4.68
+**DOGV**  | va	     | 18.598M	          |   311.380M  |    24.138M    |    2.731M     |   11.416M    |   338.250M   |         18.19           |      4.88
+**DSCCV** | va	     | 2.353M	          |   46.116M   |     0.554M    |    2.352m     |   5.031M     |   46.672M    |         19.84           |      4.56
+**DSCV**  | va	     | 1.646M	          |   32.496M   |     0.433M    |    1.427m     |   3.796M     |   32.930M    |         20.01           |      4.65
+**UN**    | va	     | 0.394M	          |   12.289M   |     0.253M    |    0.015M     |   0.533M     |   12.556M    |         31.86           |      4.86
+**VJ**    | va	     | 0.913M	          |   23.594M   |     0.466M    |    23.314m    |   0.849M     |   24.084M    |         26.39           |      4.57
+
+
+Several of the downloaded sources have already been used in the Meta-Llama-3-8B training, so the date of data collection for the previous
+model has been taken into account and those web pages have been scraped from that date.
+
+Information on the datasets used for training is shown below:
+
+- Official Bulletin of the University of Alicante (BOUA): These are the documents issued by the University of Alicante related to grants, regulations, and different resolutions of laws published periodically, specifically the Valencian version.
+
+- Legacy Official Journal of the Generalitat Valenciana (DOGCV): This journal contains historical documents issued by the Valencian Community. These documents were initially recorded on paper and digitised with the standardisation of the digital format. They have the same subject matter as the DOGV documents but were generated between 1980 and 1997.
+
+- Official Journal of the Generalitat Valenciana (DOGV): These documents contain official communications of the Valencian Community. They mainly deal with issuing laws, legal measures, and public sector communication. These journals were issued from 1998 to 2023.
+
+- Valencian Parliament Diary Dataset (DSCCV) contains records from various committee meetings held in the parliament, with each meeting documented in a separate text file.
+
+- Journal of the Valencian Parliament (DSCV): in this case, the transcripts of the different meetings held in the parliament's plenary sessions, with data from 1999 to 2022.
+
+- University news (UN): We have news in a colloquial register from different universities that have Valencian as an official language, including the universities of Valencia, Alicante, Jaume I, and the Polytechnic University of Valencia.
+
+- Valencian Journals (VJ): These include various types of Valencian journals with colloquial records to facilitate daily record-keeping alongside the legal and bureaucratic documents from previous records. These include a total of 10 different journals.
+
+
+
+### Training parameters
+
+During the training of the model, a high context window was desired when generating text, so it was decided to use an input size of 2048
+tokens and a minimum context window of 512 in case of truncating the input sequences. 80% of the data obtained was used for the training stage,
+while 20% was used during the evaluation stage. A summary of the parameters used during training can be seen in the following table:
+
+Parameter	                    | Value	| 
+|-------------------------------|-------|
+Epochs                          |    2  |
+Learning Rate                   |  2e-5 |
+Warmup Steps                    |   0   |
+Precision                       | bf-16 |
+Weight decay                    | 1e-1  |
+Training Fraction               | 0.95  |
+Evaluation Fraction             | 0.05  |
+Input size (tokens)	            | 2048	|
+Minimum context window (tokens) |  512  |
+
+
+
+### Distributed Training Strategy
+A distributed training strategy called Fully Sharded Data Parallel (FSDP)
+has been used. With this, the entire model has been loaded among the 4 A100s available for training with a mini-batch size of size 1 and a total gradient accumulation step of 64.
+
+### Languages
+In addition to the data already used for the training of Meta-Llama-3-8B, data completely in Valencian from the sources mentioned in
+the previous section has been used.
+
+## Evaluation
+
+In the following table, we can see the results obtained with different benchmarks in comparison with
+the model used for continuous pre-training. The results have been obtained from the model pre-trained;
+no instruction tuning or fine-tuning of any kind has been performed.
+
+
+### Valencian
+
+#### Classification Benchmarks
+
+| Dataset                      |  Lang. |          Task              | Metric      | Salamandra-2B |  Aitana-2B-S      |
+|------------------------------|--------|----------------------------|-------------|-----------|-----------------------|
+| XNLI                         |   va   |Natural Language Inference  |     acc     |   **0.475**   |         0.473     |
+
+
+
+#### Generation Benchmarks
+
+| Dataset                      |  Lang. |          Task              | Metric      | Salamandra-2B |  Aitana-2B-S          |
+|------------------------------|--------|----------------------------|-------------|---------------|-----------------------|
+| Cocoteros                    |   va   |Reading Comprehension       |     bleu    |   **6.32**    |         5.76          |
+| Phrases ca-va                |  va-ca |Translation - Adaptation    |     bleu    |   79.82       |         **81.92**     |
+| Phrases va-ca                |  va-ca |Translation - Adaptation    |     bleu    |   **78.05**   |         76.53         |
+| Phrases va-es                |  va-es |Translation                 |     bleu    |   **76.04**   |         75.99         |
+| Phrases es-va                |  es-va |Translation                 |     bleu    |   58.86       |         **61.51**     |
+
+
+
+
+
+### Catalan
+
+#### Classification Benchmarks
+ 
+| Dataset                      |  Lang. |          Task             | Metric      | Salamandra-2B |  Aitana-2B-S      | 
+|------------------------------|--------|---------------------------|-------------|---------------|-------------------|
+| Belebele Cat_latn            |   ca   | Reading Comprehension     |     acc     |   0.231       |   **0.257**       |
+| COPA                         |   ca   | Commonsense Reasoning     |     acc     |   0.700       |   **0.712**       |
+| XStoryCloze                  |   ca   | Commonsense Reasoning     |     acc     |   0.655       |   **0.657**       |
+| OpenBookQA                   |   ca   | Question Answering        |     acc     |   **0.294**   |       0.282       |
+| PAWS                         |   ca   | Paraphrasing              |     acc     |   **0.556**   |       0.551       |
+| PiQA                         |   ca   | Question Answering        |     acc     |   0.643       |   **0.646**       |
+| SiQA                         |   ca   | Question Answering        |     acc     |   **0.434**   |       0.432       |
+| ARC Easy                     |   ca   | Question Answering        |     acc     |   **0.551**   |       0.549       |
+| ARC Challenge                |   ca   | Question Answering        |     acc     |   **0.290**   |       0.288       |
+| XNLI                         |   ca   | Natural Language Inference|     acc     |   0.473       |   **0.480**       |
+| Teca                         |   ca   | Natural Language Inference|     acc     |   **0.465**   |       0.459       |
+| WNLI                         |   ca   | Natural Language Inference|     acc     |   **0.577**   |       0.563       |
+| Catcola                      |   ca   | Linguistic Acceptability  |     acc     |   **0.543**   |       0.525       |
+| Catcola                      |   ca   | Linguistic Acceptability  |     mcc     |   **0.046**   |       0.023       |
+| Catalanqa                    |   ca   | Question Answering        |      F1     |   **0.668**   |       0.655       |
+| Mgsm direct                  |   ca   | Math                      | exact match |   0.024       |   **0.028**       |
+| Catalanqa                    |   ca   | Question Answering        | exact match |   **0.437**   |       0.415       |
+| Xquad                        |   ca   | Question Answering        | exact match |   **0.371**   |       0.354       |
+| Xquad                        |   ca   | Question Answering        |      F1     |   **0.579**   |       0.566       |
+
+
+
+
+#### Generation Benchmarks
+
+| Dataset                      |  Lang. |          Task            | Metric |  Salamandra-2B |  Aitana-2B-S          |
+|------------------------------|--------|--------------------------|--------|----------------|-----------------------|
+| Cabreu abstractive           |  ca    | Summarization            |  bleu  |    5.78        |         **6.24**      |
+| Cabreu extractive            |  ca    | Summarization            |  bleu  |    **42.89**   |         41.19         |
+| Cabreu extreme               |  ca    | Summarization            |  bleu  |    3.29        |         **3.81**      |
+
+### Spanish
+
+
+#### Classification Benchmarks
+
+| Dataset                      |  Lang. |          Task             |    Metric   | Salamandra-2B |  Aitana-2B-S  |
+|------------------------------|--------|---------------------------|-------------|-----------|-----------------------|
+| Belebele Cat_latn            |   es   | Reading Comprehension     |     acc     |   **0.228**   |        0.224          |
+| PAWS                         |   es   | Paraphrasing              |     acc     |   **0.561**   |        0.543          |
+| XNLI                         |   es   | Natural Language Inference|     acc     |   **0.439**   |        0.422          |
+| WNLI                         |   es   | Natural Language Inference|     acc     |   0.563   |        0.563          |
+| XStoryCloze                  |   es   | Commonsense Reasoning     |     acc     |   **0.653**   |        0.652          |
+| Escola                       |   es   | Linguistic Acceptability  |     acc     |   **0.593**   |        0.536          |
+| Escola                       |   es   | Linguistic Acceptability  |     mcc     |   **0.031**   |        0.010          | 
+| OpenbookQA                   |   es   | Question Answering        |     acc     |   0.308   |        **0.314**          |
+| MGSM Direct                  |   es   | Math                      | exact match |   0.020   |        0.020          |
+| XQUAD                        |   es   | Question Answering        | exact match |   **0.377**   |        0.373          |
+| XQUAD                        |   es   | Question Answering        |      F1     |   **0.584**   |        0.583          |
+       
+
+
+
+#### Generation Benchmarks
+
+| Dataset                      |  Lang. |      Task      | Metric  |  Salamandra-2B  |  Aitana-2B-S  |
+|------------------------------|--------|----------------|---------|-------------|-----------------------|
+| Cocoteros                    |   es   |Reading Comprehension|  bleu   |    **8.46**    |  7.35         |
+| XLSum                        |   es   | Summarization  |  bleu   |    **0.801**    |         0.434 |
+
+
+
+### English
+
+
+#### Classification Benchmarks
+
+| Dataset                      |  Lang. |          Task              | Metric      | Salamandra-2B |  Aitana-2B-S  |
+|------------------------------|--------|----------------------------|-------------|-----------|-----------------------|
+| Arc Challenge                |   en   | Question Answering         |     acc     |   0.370   |         **0.374**         |
+| Arc Easy                     |   en   | Question Answering         |     acc     |   **0.722**   |         0.719         |
+| Belebele Eng_latn            |   en   |   Reading Comprehension    |     acc     |   0.216   |         **0.229**        |
+| PAWS                         |   en   |      Paraphrasing          |     acc     |   0.561   |         **0.562**         |
+| XNLI                         |   en   | Natural Language Inference |     acc     |   **0.462**   |         0.446         | 
+| XStoryCloze                  |   en   |   Commonsense Reasoning    |     acc     |   0.711   |         **0.713**         |
+| OpenBookQA                   |   en   |    Question Answering      |     acc     |   0.300   |         **0.308**         |
+| PiQA                         |   en   |    Question Answering      |     acc     |   0.737   |         **0.743**         |
+| Social iqa                   |   en   |    Question Answering      |     acc     |   **0.454**   |         0.451         |
+| WNLI                         |   en   |    Natural Language Inference |     acc     |   0.465   |         **0.578**         |
+| MGSM Direct                  |   en   |           Math             | exact match |   0.064   |         0.064         |
+| TriviaQA                     |   en   |     Question Answering     | exact match |   -0.019  |         **0.015**         |
+
+
+
+
+## Additional information
+
+
+### Author
+
+Language and Information System Group GPLSI
+
+
+### Contact
+
+For further information, please send an email to GPLSI
+
+
+
+### Copyright
+
+Copyright(c) 2025 by GPLSI(https://gplsi.dlsi.ua.es/).
+
+
+### License
+
+[Apache License 2.0](https://www.apache.org/licenses/LICENSE-2.0)
+
+
+### Funding
+This work was funded by [ILENIA](https://proyectoilenia.es/)-[VIVES](https://vives.gplsi.es/) project <<2022/TL22/00215334>>
+
+
+### Disclaimer
+
+The model published in this repository is intended for a generalist purpose and is available to third parties under a permissive Apache License, Version 2.0.
+
+Be aware that the model may have biases and/or any other undesirable distortions.
+
+When third parties deploy or provide systems and/or services to other parties using this model (or any system based on it) or become users of the model, they should note that it is their responsibility to mitigate the risks arising from its use and, in any event, to comply with applicable regulations, including regulations regarding the use of Artificial Intelligence.
+
+In no event shall the owner and creator of the model (GPLSI) be liable for any results arising from the use made by third parties.

config.json

@@ -0,0 +1,30 @@
+{
+  "_name_or_path": "BSC-LT/salamandra-2b",
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "head_dim": 128,
+  "hidden_act": "silu",
+  "hidden_size": 2048,
+  "initializer_range": 0.02,
+  "intermediate_size": 5440,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 16,
+  "num_hidden_layers": 24,
+  "num_key_value_heads": 16,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_scaling": null,
+  "rope_theta": 10000.0,
+  "tie_word_embeddings": false,
+  "torch_dtype": "bfloat16",
+  "transformers_version": "4.44.2",
+  "use_cache": false,
+  "vocab_size": 256000
+}

generation_config.json

@@ -0,0 +1,11 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "do_sample": true,
+  "temperature": 0.1,
+  "top_p": 0.95,
+  "max_new_tokens": 40,
+  "repetition_penalty": 1.2,
+  "transformers_version": "4.44.2"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2161af68419ee601de3bf72f0fe1fb532cee2deabb71f2e745a78e75b2e22a1b
+size 4507005744

special_tokens_map.json

@@ -0,0 +1,24 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": "</s>",
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c205db342a8a7321a4147629d806caac1c70a8cb1cebf4c0f3636dec7a3452e0
+size 19092405

tokenizer.model

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ab94ddf46d14f0279254858d53770c5319c5129d47291ee2bada530271cb1292
+size 4813276

tokenizer_config.json

@@ -0,0 +1,64 @@
+{
+  "add_bos_token": true,
+  "add_eos_token": false,
+  "add_prefix_space": true,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "4": {
+      "content": "<|im_start|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "5": {
+      "content": "<|im_end|>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "additional_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "bos_token": "<s>",
+  "chat_template": "{%- if not date_string is defined %}{%- set date_string = \"2025-06-30\" %}{%- endif %}{%- set system_message = messages[0].content if messages[0].role == \"system\" else \"I am Aitana, experimental model developed at the University of Alicante by the Language and Information Systems Group (GPLSI). My knowledge base was last updated on May 2025. Today Date: \"+ date_string +\"\nSoy Aitana, un modelo experimental desarrollado en la Universidad de Alicante por el Grupo de Procesamiento del Lenguaje y Sistemas de Información (GPLSI). Mi base de conocimiento se actualizó por última vez en mayo de 2025.\nSoc Aitana, un model experimental desenvolupat a la Universitat d'Alacant pel Grup de Processament del Llenguatge i Sistemes d'Informació (GPLSI). La meua base de coneixement es va actualitzar per última vegada en maig del 2025.\" -%}{%- if messages[0].role == \"system\" -%}{%- set messages = messages[1:] -%}{%- endif -%}{{ \"<|im_start|>system\n\" + system_message + \"<|im_end|>\n\" }}{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "</s>",
+  "legacy": true,
+  "model_max_length": 8192,
+  "pad_token": "<unk>",
+  "padding_side": "right",
+  "sp_model_kwargs": {},
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "LlamaTokenizer",
+  "unk_token": "<unk>",
+  "use_default_system_prompt": false
+}