mediabiasgroup/anno-lexical-mdeberta

mDeBERTa-SA-FT (Anno-lexical)

This model is a sentence-level media (lexical) bias classifier trained on the Anno-lexical dataset from
“The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection” (NAACL Findings 2025; arXiv:2411.11081).
It uses an mdeberta-v3-base encoder with a standard sequence classification head. Labels: 0 = neutral/non-lexical-bias, 1 = lexical-bias.

Paper: The Promises and Pitfalls of LLM Annotations in Dataset Labeling: a Case Study on Media Bias Detection
Dataset: mediabiasgroup/anno-lexical

Intended use & limitations

• Intended use: research on lexical/loaded-language bias; backbone comparison (RoBERTa vs mDeBERTa).
• Out-of-scope: detection of non-lexical media bias forms (e.g., informational/selection bias), leaning, stance, factuality.
• Note: Add evaluation metrics below once computed; then include a model-index block if desired.

How to use

from transformers import AutoTokenizer, AutoModelForSequenceClassification

m = "mediabiasgroup/anno-lexical-mdeberta"
tok = AutoTokenizer.from_pretrained(m)
model = AutoModelForSequenceClassification.from_pretrained(m)

Training data & setup

Training data: Anno-lexical (binary lexical-bias labels aggregated from LLM annotations).

Base encoder: mdeberta-v3-base; head: 2-layer classifier.

Hardware: single A100; single-run training.

Evaluation

TBD — run the standard evaluation suite (e.g., BABE test, BASIL sentences) and add metrics here. After that, create a model-index with the results to enable the HF results widget.

Safety, bias & ethics

Media bias perception is subjective and culturally dependent. This model may over-flag biased wording and should not be used to penalize individuals or outlets. Use with human-in-the-loop review and domain-specific calibration.

Citation

If you use this model, please cite:

@inproceedings{horych-etal-2025-promises,
  title = "The Promises and Pitfalls of {LLM} Annotations in Dataset Labeling: a Case Study on Media Bias Detection",
  author = "Horych, Tom{\'a}{\v{s}}  and
    Mandl, Christoph  and
    Ruas, Terry  and
    Greiner-Petter, Andre  and
    Gipp, Bela  and
    Aizawa, Akiko  and
    Spinde, Timo",
  editor = "Chiruzzo, Luis  and
    Ritter, Alan  and
    Wang, Lu",
  booktitle = "Findings of the Association for Computational Linguistics: NAACL 2025",
  month = apr,
  year = "2025",
  address = "Albuquerque, New Mexico",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2025.findings-naacl.75/",
  doi = "10.18653/v1/2025.findings-naacl.75",
  pages = "1370--1386",
  ISBN = "979-8-89176-195-7"
}