File size: 6,407 Bytes
ba7967b 17f0c9a ba7967b 17f0c9a ba7967b |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 |
---
license: mit
base_model: vinai/bertweet-base
tags:
- generated_from_trainer
metrics:
- f1
- precision
- recall
- accuracy
model-index:
- name: bertweet-base_ordinal_7_seed42_EN-NL
results: []
---
<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->
# bertweet-base_ordinal_7_seed42_EN-NL
This model is a fine-tuned version of [vinai/bertweet-base](https://huggingface.co/vinai/bertweet-base) on an unknown dataset.
It achieves the following results on the evaluation set:
- Loss: 3.9585
- Mse: 6.1125
- Rmse: 2.4723
- Mae: 1.5109
- R2: 0.1525
- F1: 0.7447
- Precision: 0.7446
- Recall: 0.7479
- Accuracy: 0.7479
## Model description
This is the best-performing model for Dutch irony detection. The model was fine-tuned both a mix of English and Dutch tweets.
The model predicts one of 7 labels indicating for irony likelihood, where 0 is not ironic and 6 is ironic.
When merging for binary classification, we advise mapping labels 0,1,2,3 as not-ironic and labels 4,5,6 as ironic.
## Intended uses & limitations
More information needed
## Training and evaluation data
The model was trained and evaluated on the TRIC dataset.
## Training procedure
### Training hyperparameters
The following hyperparameters were used during training:
- learning_rate: 5e-06
- train_batch_size: 16
- eval_batch_size: 16
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- lr_scheduler_warmup_steps: 200
- num_epochs: 10
### Training results
| Training Loss | Epoch | Step | Validation Loss | Mse | Rmse | Mae | R2 | F1 | Precision | Recall | Accuracy |
|:-------------:|:------:|:----:|:---------------:|:------:|:------:|:------:|:-------:|:------:|:---------:|:------:|:--------:|
| 5.7557 | 0.2141 | 100 | 5.6476 | 7.6297 | 2.7622 | 2.6574 | -0.0806 | 0.4761 | 0.3859 | 0.6212 | 0.6212 |
| 5.2388 | 0.4283 | 200 | 5.2492 | 7.1086 | 2.6662 | 2.4741 | -0.0068 | 0.4761 | 0.3859 | 0.6212 | 0.6212 |
| 4.9773 | 0.6424 | 300 | 5.0558 | 6.8733 | 2.6217 | 2.3016 | 0.0266 | 0.4761 | 0.3859 | 0.6212 | 0.6212 |
| 4.7427 | 0.8565 | 400 | 4.8666 | 6.6212 | 2.5732 | 2.1990 | 0.0623 | 0.4761 | 0.3859 | 0.6212 | 0.6212 |
| 4.6378 | 1.0707 | 500 | 4.6806 | 6.0772 | 2.4652 | 2.0941 | 0.1393 | 0.6773 | 0.6795 | 0.6888 | 0.6888 |
| 4.3851 | 1.2848 | 600 | 4.6153 | 6.2799 | 2.5060 | 1.9928 | 0.1106 | 0.6915 | 0.6964 | 0.6888 | 0.6888 |
| 4.3077 | 1.4989 | 700 | 4.5016 | 6.2147 | 2.4929 | 1.9276 | 0.1198 | 0.6882 | 0.6928 | 0.7008 | 0.7008 |
| 4.2337 | 1.7131 | 800 | 4.2877 | 5.5862 | 2.3635 | 1.8854 | 0.2088 | 0.7183 | 0.7218 | 0.7274 | 0.7274 |
| 4.2273 | 1.9272 | 900 | 4.3769 | 5.9397 | 2.4371 | 1.8601 | 0.1588 | 0.6994 | 0.6991 | 0.6996 | 0.6996 |
| 4.0563 | 2.1413 | 1000 | 4.2168 | 6.1013 | 2.4701 | 1.7033 | 0.1359 | 0.7088 | 0.7203 | 0.7238 | 0.7238 |
| 3.7778 | 2.3555 | 1100 | 4.1356 | 6.1098 | 2.4718 | 1.6562 | 0.1347 | 0.7260 | 0.7269 | 0.7322 | 0.7322 |
| 3.7206 | 2.5696 | 1200 | 4.2222 | 6.1062 | 2.4711 | 1.7394 | 0.1352 | 0.7245 | 0.7326 | 0.7214 | 0.7214 |
| 3.7175 | 2.7837 | 1300 | 4.0073 | 5.7021 | 2.3879 | 1.6224 | 0.1924 | 0.7277 | 0.7345 | 0.7382 | 0.7382 |
| 3.8003 | 2.9979 | 1400 | 4.1116 | 5.8166 | 2.4118 | 1.7346 | 0.1762 | 0.7258 | 0.7268 | 0.7250 | 0.7250 |
| 3.6247 | 3.2120 | 1500 | 4.1286 | 6.0663 | 2.4630 | 1.6876 | 0.1409 | 0.7309 | 0.7355 | 0.7286 | 0.7286 |
| 3.4364 | 3.4261 | 1600 | 4.2100 | 6.3353 | 2.5170 | 1.7467 | 0.1028 | 0.7235 | 0.7329 | 0.7201 | 0.7201 |
| 3.3301 | 3.6403 | 1700 | 4.0403 | 6.0483 | 2.4593 | 1.6357 | 0.1434 | 0.7436 | 0.7442 | 0.7431 | 0.7431 |
| 3.3634 | 3.8544 | 1800 | 3.9496 | 5.5790 | 2.3620 | 1.6297 | 0.2099 | 0.7259 | 0.7282 | 0.7334 | 0.7334 |
| 3.4602 | 4.0685 | 1900 | 3.8729 | 5.7334 | 2.3945 | 1.5597 | 0.1880 | 0.7402 | 0.7410 | 0.7455 | 0.7455 |
| 3.1223 | 4.2827 | 2000 | 4.0417 | 6.3812 | 2.5261 | 1.5875 | 0.0963 | 0.7144 | 0.7394 | 0.7346 | 0.7346 |
| 3.1337 | 4.4968 | 2100 | 4.0039 | 5.9493 | 2.4391 | 1.6285 | 0.1574 | 0.7389 | 0.7421 | 0.7370 | 0.7370 |
| 3.1321 | 4.7109 | 2200 | 3.9092 | 5.8926 | 2.4275 | 1.5742 | 0.1655 | 0.7347 | 0.7339 | 0.7358 | 0.7358 |
| 3.1927 | 4.9251 | 2300 | 4.0312 | 5.9928 | 2.4480 | 1.6140 | 0.1513 | 0.7459 | 0.7540 | 0.7431 | 0.7431 |
| 2.9806 | 5.1392 | 2400 | 3.9638 | 6.0145 | 2.4524 | 1.5633 | 0.1482 | 0.7524 | 0.7536 | 0.7515 | 0.7515 |
| 2.9582 | 5.3533 | 2500 | 3.9413 | 5.9409 | 2.4374 | 1.5549 | 0.1586 | 0.7539 | 0.7539 | 0.7539 | 0.7539 |
| 2.7418 | 5.5675 | 2600 | 3.9578 | 5.9843 | 2.4463 | 1.5525 | 0.1525 | 0.7456 | 0.7476 | 0.7443 | 0.7443 |
| 2.9866 | 5.7816 | 2700 | 3.8793 | 5.8070 | 2.4098 | 1.5416 | 0.1776 | 0.7426 | 0.7425 | 0.7467 | 0.7467 |
| 2.8627 | 5.9957 | 2800 | 3.8625 | 5.7805 | 2.4043 | 1.5103 | 0.1813 | 0.7615 | 0.7609 | 0.7624 | 0.7624 |
| 2.8191 | 6.2099 | 2900 | 3.9342 | 5.9964 | 2.4488 | 1.5211 | 0.1508 | 0.7628 | 0.7622 | 0.7636 | 0.7636 |
| 2.6259 | 6.4240 | 3000 | 3.9203 | 6.0893 | 2.4676 | 1.5006 | 0.1376 | 0.7487 | 0.7478 | 0.7503 | 0.7503 |
| 2.8785 | 6.6381 | 3100 | 3.8633 | 5.8444 | 2.4175 | 1.4946 | 0.1723 | 0.7600 | 0.7601 | 0.7600 | 0.7600 |
| 2.6016 | 6.8522 | 3200 | 4.0736 | 6.2654 | 2.5031 | 1.5923 | 0.1127 | 0.7456 | 0.7518 | 0.7431 | 0.7431 |
| 2.5155 | 7.0664 | 3300 | 3.9459 | 6.0688 | 2.4635 | 1.5211 | 0.1405 | 0.7584 | 0.7597 | 0.7575 | 0.7575 |
| 2.6918 | 7.2805 | 3400 | 3.9312 | 6.0072 | 2.4510 | 1.5271 | 0.1492 | 0.7541 | 0.7534 | 0.7551 | 0.7551 |
### Framework versions
- Transformers 4.40.2
- Pytorch 2.1.2
- Datasets 2.18.0
- Tokenizers 0.19.1
|