Improve model card: Add pipeline tag, update metadata, and enrich content
#1
by
nielsr
HF Staff
- opened
README.md
CHANGED
@@ -1,23 +1,43 @@
|
|
1 |
---
|
2 |
-
library_name: transformers
|
3 |
-
license: mit
|
4 |
base_model: microsoft/mdeberta-v3-base
|
5 |
-
|
6 |
-
-
|
7 |
metrics:
|
8 |
- accuracy
|
9 |
- f1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
10 |
model-index:
|
11 |
- name: mdeberta-v3-base-subjectivity-multilingual
|
12 |
results: []
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
13 |
---
|
14 |
|
15 |
-
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
|
16 |
-
should probably proofread and complete it, then remove this comment. -->
|
17 |
-
|
18 |
# mdeberta-v3-base-subjectivity-multilingual
|
19 |
|
20 |
-
This model is a fine-tuned version of [microsoft/mdeberta-v3-base](https://huggingface.co/microsoft/mdeberta-v3-base)
|
|
|
|
|
|
|
21 |
It achieves the following results on the evaluation set:
|
22 |
- Loss: 0.8345
|
23 |
- Macro F1: 0.7475
|
@@ -30,15 +50,59 @@ It achieves the following results on the evaluation set:
|
|
30 |
|
31 |
## Model description
|
32 |
|
33 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
34 |
|
35 |
## Intended uses & limitations
|
36 |
|
37 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
38 |
|
39 |
## Training and evaluation data
|
40 |
|
41 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
42 |
|
43 |
## Training procedure
|
44 |
|
@@ -62,7 +126,7 @@ The following hyperparameters were used during training:
|
|
62 |
| 0.4631 | 3.0 | 1206 | 0.6583 | 0.7328 | 0.7311 | 0.7353 | 0.6785 | 0.6609 | 0.6971 | 0.7439 |
|
63 |
| 0.394 | 4.0 | 1608 | 0.7692 | 0.7255 | 0.7327 | 0.7215 | 0.6523 | 0.6924 | 0.6165 | 0.7451 |
|
64 |
| 0.3475 | 5.0 | 2010 | 0.7538 | 0.7438 | 0.7414 | 0.7481 | 0.6951 | 0.6667 | 0.7261 | 0.7530 |
|
65 |
-
| 0.3475 | 6.0 | 2412 | 0.8345 | 0.7475 | 0.7530 | 0.7439 | 0.6824
|
66 |
|
67 |
|
68 |
### Framework versions
|
@@ -70,4 +134,24 @@ The following hyperparameters were used during training:
|
|
70 |
- Transformers 4.49.0
|
71 |
- Pytorch 2.5.1+cu121
|
72 |
- Datasets 3.3.1
|
73 |
-
- Tokenizers 0.21.0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
|
|
|
|
2 |
base_model: microsoft/mdeberta-v3-base
|
3 |
+
library_name: transformers
|
4 |
+
license: cc-by-4.0
|
5 |
metrics:
|
6 |
- accuracy
|
7 |
- f1
|
8 |
+
tags:
|
9 |
+
- generated_from_trainer
|
10 |
+
- deberta-v3
|
11 |
+
- subjectivity-detection
|
12 |
+
- multilingual
|
13 |
+
- text-classification
|
14 |
+
- sentiment-analysis
|
15 |
model-index:
|
16 |
- name: mdeberta-v3-base-subjectivity-multilingual
|
17 |
results: []
|
18 |
+
pipeline_tag: text-classification
|
19 |
+
language:
|
20 |
+
- ar
|
21 |
+
- de
|
22 |
+
- en
|
23 |
+
- it
|
24 |
+
- bg
|
25 |
+
- el
|
26 |
+
- pl
|
27 |
+
- ro
|
28 |
+
- uk
|
29 |
+
datasets:
|
30 |
+
- MatteoFasulo/clef2025_checkthat_task1_subjectivity
|
31 |
+
arxiv_id: 2507.11764
|
32 |
+
code_url: https://github.com/MatteoFasulo/clef2025-checkthat
|
33 |
---
|
34 |
|
|
|
|
|
|
|
35 |
# mdeberta-v3-base-subjectivity-multilingual
|
36 |
|
37 |
+
This model is a fine-tuned version of [microsoft/mdeberta-v3-base](https://huggingface.co/microsoft/mdeberta-v3-base) for **Subjectivity Detection in News Articles**. It was developed as part of **AI Wizards' participation in the CLEF 2025 CheckThat! Lab Task 1**.
|
38 |
+
|
39 |
+
The model was presented in the paper [AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles](https://huggingface.co/papers/2507.11764).
|
40 |
+
|
41 |
It achieves the following results on the evaluation set:
|
42 |
- Loss: 0.8345
|
43 |
- Macro F1: 0.7475
|
|
|
50 |
|
51 |
## Model description
|
52 |
|
53 |
+
This model, `mdeberta-v3-base-subjectivity-multilingual`, is designed to classify sentences as subjective (opinion-laden) or objective (fact-based) within news articles. It was developed by AI Wizards for the CLEF 2025 CheckThat! Lab Task 1: Subjectivity Detection in News Articles.
|
54 |
+
|
55 |
+
The core innovation of this model lies in enhancing standard transformer-based classifiers by integrating sentiment scores, derived from an auxiliary model, with sentence representations. This sentiment-augmented architecture, built upon mDeBERTaV3-base, aims to significantly improve performance, particularly for the subjective F1 score. To counteract prevalent class imbalance across languages, decision threshold calibration optimized on the development set was employed.
|
56 |
+
|
57 |
+
The model was evaluated across:
|
58 |
+
* **Monolingual** settings (Arabic, German, English, Italian, and Bulgarian)
|
59 |
+
* **Zero-shot transfer** settings (Greek, Polish, Romanian, and Ukrainian)
|
60 |
+
* **Multilingual** training
|
61 |
+
|
62 |
+
This framework led to high rankings in the competition, notably achieving 1st place for Greek (Macro F1 = 0.51).
|
63 |
|
64 |
## Intended uses & limitations
|
65 |
|
66 |
+
**Intended uses:**
|
67 |
+
This model is intended for research and practical applications involving subjectivity detection, particularly in news media. Specific uses include:
|
68 |
+
* Classifying sentences in news articles as subjective or objective.
|
69 |
+
* Supporting fact-checking pipelines by identifying opinionated content.
|
70 |
+
* Assisting journalists in analyzing text for bias or subjective reporting.
|
71 |
+
* Applications in both monolingual and multilingual contexts, including zero-shot scenarios for unseen languages.
|
72 |
+
|
73 |
+
**Limitations:**
|
74 |
+
* Performance may vary across different languages, especially in zero-shot settings, despite efforts for generalization.
|
75 |
+
* The effectiveness of the sentiment augmentation relies on the quality and domain relevance of the auxiliary sentiment model.
|
76 |
+
* While designed for news articles, its performance might differ on other text genres or domains.
|
77 |
+
* Like other large language models, it may carry biases present in its training data.
|
78 |
|
79 |
## Training and evaluation data
|
80 |
|
81 |
+
The model was fine-tuned on training and development datasets provided for the CLEF 2025 CheckThat! Lab Task 1. These datasets included sentences from news articles in Arabic, German, English, Italian, and Bulgarian. For final evaluation, additional unseen languages such as Greek, Romanian, Polish, and Ukrainian were included to assess the model's generalization capabilities. Class imbalance issues, which were prevalent across languages, were addressed through decision threshold calibration.
|
82 |
+
|
83 |
+
## How to use
|
84 |
+
|
85 |
+
You can easily use this model with the Hugging Face `transformers` library:
|
86 |
+
|
87 |
+
```python
|
88 |
+
from transformers import pipeline
|
89 |
+
|
90 |
+
# Load the text classification pipeline
|
91 |
+
classifier = pipeline(
|
92 |
+
"text-classification",
|
93 |
+
model="MatteoFasulo/mdeberta-v3-base-subjectivity-multilingual",
|
94 |
+
tokenizer="microsoft/mdeberta-v3-base",
|
95 |
+
)
|
96 |
+
|
97 |
+
# Example usage:
|
98 |
+
result1 = classifier("Questa è una scoperta affascinante e fantastica!")
|
99 |
+
print(f"Classification: {result1}")
|
100 |
+
# Expected output: [{'label': 'SUBJ', 'score': ...}]
|
101 |
+
|
102 |
+
result2 = classifier("The capital of France is Paris.")
|
103 |
+
print(f"Classification: {result2}")
|
104 |
+
# Expected output: [{'label': 'OBJ', 'score': ...}]
|
105 |
+
```
|
106 |
|
107 |
## Training procedure
|
108 |
|
|
|
126 |
| 0.4631 | 3.0 | 1206 | 0.6583 | 0.7328 | 0.7311 | 0.7353 | 0.6785 | 0.6609 | 0.6971 | 0.7439 |
|
127 |
| 0.394 | 4.0 | 1608 | 0.7692 | 0.7255 | 0.7327 | 0.7215 | 0.6523 | 0.6924 | 0.6165 | 0.7451 |
|
128 |
| 0.3475 | 5.0 | 2010 | 0.7538 | 0.7438 | 0.7414 | 0.7481 | 0.6951 | 0.6667 | 0.7261 | 0.7530 |
|
129 |
+
| 0.3475 | 6.0 | 2412 | 0.8345 | 0.7475 | 0.7530 | 0.7439 | 0.6824 | 0.7145 | 0.6531 | 0.7643 |
|
130 |
|
131 |
|
132 |
### Framework versions
|
|
|
134 |
- Transformers 4.49.0
|
135 |
- Pytorch 2.5.1+cu121
|
136 |
- Datasets 3.3.1
|
137 |
+
- Tokenizers 0.21.0
|
138 |
+
|
139 |
+
## GitHub Repository
|
140 |
+
|
141 |
+
The code and materials for this model are available on GitHub: [MatteoFasulo/clef2025-checkthat](https://github.com/MatteoFasulo/clef2025-checkthat)
|
142 |
+
|
143 |
+
## Citation
|
144 |
+
|
145 |
+
If you find our work helpful or inspiring, please feel free to cite it:
|
146 |
+
|
147 |
+
```bibtex
|
148 |
+
@misc{fasulo2025aiwizardscheckthat2025,
|
149 |
+
title={AI Wizards at CheckThat! 2025: Enhancing Transformer-Based Embeddings with Sentiment for Subjectivity Detection in News Articles},
|
150 |
+
author={Matteo Fasulo and Luca Babboni and Luca Tedeschini},
|
151 |
+
year={2025},
|
152 |
+
eprint={2507.11764},
|
153 |
+
archivePrefix={arXiv},
|
154 |
+
primaryClass={cs.CL},
|
155 |
+
url={https://arxiv.org/abs/2507.11764},
|
156 |
+
}
|
157 |
+
```
|