|
|
--- |
|
|
language: |
|
|
- en |
|
|
base_model: |
|
|
- answerdotai/ModernBERT-large |
|
|
pipeline_tag: text-classification |
|
|
tags: |
|
|
- GEC |
|
|
--- |
|
|
|
|
|
# IMPARA-GED |
|
|
|
|
|
This is the repository of the IMPARA-GED, a quality estimator for the grammatical error correction evaluation. |
|
|
|
|
|
--- |
|
|
## Abstract |
|
|
From [IMPARA-GED: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator](https://aclanthology.org/2025.findings-acl.1315/) (Findings of ACL2025) |
|
|
|
|
|
> We propose IMPARA-GED, a novel reference-free automatic grammatical error correction (GEC) evaluation method with grammatical error detection (GED) capabilities. We focus on the quality estimator of IMPARA, an existing automatic GEC evaluation method, and construct that of IMPARA-GED using a pre-trained language model with enhanced GED capabilities. Experimental results on SEEDA, a meta-evaluation dataset for automatic GEC evaluation methods, demonstrate that IMPARA-GED achieves the highest correlation with human sentence-level evaluations. |
|
|
|
|
|
--- |
|
|
## Usage |
|
|
|
|
|
STEP 1: |
|
|
``` |
|
|
pip install gec-metric |
|
|
``` |
|
|
|
|
|
STEP 2: |
|
|
``` |
|
|
from gec_metrics import get_metric, get_meta_eval |
|
|
from pprint import pprint |
|
|
c = get_metric('impara') |
|
|
metric = c(c.Config( |
|
|
model_qe='naist-nlp/IMPARA-GED', |
|
|
threshold=-999.0, # Ignore similarity scores |
|
|
pooling='mean' # Our model was trained with mean pooling |
|
|
)) |
|
|
print(metric.config) |
|
|
c = get_meta_eval('seeda') |
|
|
meta = c(c.Config('base')) |
|
|
|
|
|
print('=== System-level meta-evaluation ===') |
|
|
system_results = meta.corr_system(metric, aggregation='trueskill') |
|
|
print('SEEDA-S') |
|
|
pprint(system_results.ts_sent) |
|
|
print('SEEDA-E') |
|
|
pprint(system_results.ts_edit) |
|
|
|
|
|
print('=== Sentence-level meta-evaluation ===') |
|
|
sentence_results = meta.corr_sentence(metric) |
|
|
print('SEEDA-S') |
|
|
pprint(sentence_results.sent) |
|
|
print('SEEDA-E') |
|
|
pprint(sentence_results.edit) |
|
|
``` |
|
|
|
|
|
|
|
|
|
|
|
See details: [`gec-metric`](https://github.com/gotutiyan/gec-metrics). |
|
|
|
|
|
|
|
|
|
|
|
--- |
|
|
## Citation |
|
|
|
|
|
**Bibkey (For [anthology.bib](http://aclweb.org/anthology/anthology.bib)):** |
|
|
|
|
|
```sakai-etal-2025-impara``` |
|
|
|
|
|
Bibtex: |
|
|
``` |
|
|
@inproceedings{sakai-etal-2025-impara, |
|
|
title = "{IMPARA}-{GED}: Grammatical Error Detection is Boosting Reference-free Grammatical Error Quality Estimator", |
|
|
author = "Sakai, Yusuke and |
|
|
Goto, Takumi and |
|
|
Watanabe, Taro", |
|
|
editor = "Che, Wanxiang and |
|
|
Nabende, Joyce and |
|
|
Shutova, Ekaterina and |
|
|
Pilehvar, Mohammad Taher", |
|
|
booktitle = "Findings of the Association for Computational Linguistics: ACL 2025", |
|
|
month = jul, |
|
|
year = "2025", |
|
|
address = "Vienna, Austria", |
|
|
publisher = "Association for Computational Linguistics", |
|
|
url = "https://aclanthology.org/2025.findings-acl.1315/", |
|
|
doi = "10.18653/v1/2025.findings-acl.1315", |
|
|
pages = "25647--25654", |
|
|
ISBN = "979-8-89176-256-5", |
|
|
abstract = "We propose IMPARA-GED, a novel reference-free automatic grammatical error correction (GEC) evaluation method with grammatical error detection (GED) capabilities. We focus on the quality estimator of IMPARA, an existing automatic GEC evaluation method, and construct that of IMPARA-GED using a pre-trained language model with enhanced GED capabilities. Experimental results on SEEDA, a meta-evaluation dataset for automatic GEC evaluation methods, demonstrate that IMPARA-GED achieves the highest correlation with human sentence-level evaluations." |
|
|
} |
|
|
``` |
|
|
|
|
|
## Contact |
|
|
|
|
|
Yusuke Sakai ([@yusuke1997](https://huggingface.co/yusuke1997)) |
