adity12345 commited on
Commit
b428379
·
verified ·
1 Parent(s): c696e85

End of training

Browse files
README.md ADDED
@@ -0,0 +1,120 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: transformers
3
+ base_model: dmis-lab/biobert-base-cased-v1.1
4
+ tags:
5
+ - generated_from_trainer
6
+ metrics:
7
+ - accuracy
8
+ - precision
9
+ - recall
10
+ - f1
11
+ model-index:
12
+ - name: Biobert_combo_v1
13
+ results: []
14
+ ---
15
+
16
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
17
+ should probably proofread and complete it, then remove this comment. -->
18
+
19
+ # Biobert_combo_v1
20
+
21
+ This model is a fine-tuned version of [dmis-lab/biobert-base-cased-v1.1](https://huggingface.co/dmis-lab/biobert-base-cased-v1.1) on the None dataset.
22
+ It achieves the following results on the evaluation set:
23
+ - Loss: 0.4421
24
+ - Accuracy: 0.769
25
+ - Auc: 0.867
26
+ - Precision: 0.745
27
+ - Recall: 0.887
28
+ - F1: 0.81
29
+ - F1-macro: 0.757
30
+ - F1-micro: 0.769
31
+ - F1-weighted: 0.763
32
+
33
+ ## Model description
34
+
35
+ More information needed
36
+
37
+ ## Intended uses & limitations
38
+
39
+ More information needed
40
+
41
+ ## Training and evaluation data
42
+
43
+ More information needed
44
+
45
+ ## Training procedure
46
+
47
+ ### Training hyperparameters
48
+
49
+ The following hyperparameters were used during training:
50
+ - learning_rate: 2e-05
51
+ - train_batch_size: 8
52
+ - eval_batch_size: 8
53
+ - seed: 42
54
+ - gradient_accumulation_steps: 4
55
+ - total_train_batch_size: 32
56
+ - optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
57
+ - lr_scheduler_type: linear
58
+ - num_epochs: 5
59
+ - mixed_precision_training: Native AMP
60
+
61
+ ### Training results
62
+
63
+ | Training Loss | Epoch | Step | Validation Loss | Accuracy | Auc | Precision | Recall | F1 | F1-macro | F1-micro | F1-weighted |
64
+ |:-------------:|:------:|:-----:|:---------------:|:--------:|:-----:|:---------:|:------:|:-----:|:--------:|:--------:|:-----------:|
65
+ | 0.6198 | 0.1028 | 500 | 0.5619 | 0.693 | 0.769 | 0.676 | 0.858 | 0.757 | 0.671 | 0.693 | 0.681 |
66
+ | 0.5401 | 0.2057 | 1000 | 0.5167 | 0.717 | 0.807 | 0.717 | 0.81 | 0.761 | 0.708 | 0.717 | 0.713 |
67
+ | 0.5141 | 0.3085 | 1500 | 0.4933 | 0.731 | 0.827 | 0.75 | 0.772 | 0.761 | 0.726 | 0.731 | 0.73 |
68
+ | 0.4886 | 0.4114 | 2000 | 0.4942 | 0.736 | 0.827 | 0.727 | 0.841 | 0.78 | 0.726 | 0.736 | 0.732 |
69
+ | 0.4882 | 0.5142 | 2500 | 0.4814 | 0.743 | 0.838 | 0.712 | 0.901 | 0.795 | 0.724 | 0.743 | 0.732 |
70
+ | 0.4734 | 0.6170 | 3000 | 0.4718 | 0.746 | 0.843 | 0.724 | 0.878 | 0.794 | 0.733 | 0.746 | 0.739 |
71
+ | 0.4659 | 0.7199 | 3500 | 0.4767 | 0.748 | 0.843 | 0.733 | 0.859 | 0.791 | 0.737 | 0.748 | 0.743 |
72
+ | 0.4632 | 0.8227 | 4000 | 0.4617 | 0.755 | 0.852 | 0.724 | 0.901 | 0.803 | 0.739 | 0.755 | 0.746 |
73
+ | 0.4602 | 0.9255 | 4500 | 0.4639 | 0.752 | 0.852 | 0.717 | 0.915 | 0.804 | 0.734 | 0.752 | 0.742 |
74
+ | 0.457 | 1.0284 | 5000 | 0.4613 | 0.752 | 0.85 | 0.729 | 0.881 | 0.798 | 0.739 | 0.752 | 0.745 |
75
+ | 0.4468 | 1.1312 | 5500 | 0.4541 | 0.757 | 0.855 | 0.731 | 0.891 | 0.803 | 0.743 | 0.757 | 0.75 |
76
+ | 0.4421 | 1.2341 | 6000 | 0.4591 | 0.755 | 0.853 | 0.727 | 0.892 | 0.801 | 0.74 | 0.755 | 0.747 |
77
+ | 0.4373 | 1.3369 | 6500 | 0.4537 | 0.759 | 0.856 | 0.739 | 0.874 | 0.801 | 0.748 | 0.759 | 0.753 |
78
+ | 0.4402 | 1.4397 | 7000 | 0.4552 | 0.755 | 0.855 | 0.74 | 0.863 | 0.797 | 0.745 | 0.755 | 0.751 |
79
+ | 0.4296 | 1.5426 | 7500 | 0.4545 | 0.76 | 0.857 | 0.742 | 0.87 | 0.801 | 0.749 | 0.76 | 0.755 |
80
+ | 0.4407 | 1.6454 | 8000 | 0.4458 | 0.762 | 0.86 | 0.742 | 0.877 | 0.804 | 0.751 | 0.762 | 0.757 |
81
+ | 0.4225 | 1.7483 | 8500 | 0.4472 | 0.761 | 0.86 | 0.735 | 0.889 | 0.805 | 0.748 | 0.761 | 0.754 |
82
+ | 0.4327 | 1.8511 | 9000 | 0.4485 | 0.758 | 0.858 | 0.741 | 0.867 | 0.799 | 0.747 | 0.758 | 0.753 |
83
+ | 0.4311 | 1.9539 | 9500 | 0.4479 | 0.76 | 0.859 | 0.742 | 0.869 | 0.801 | 0.749 | 0.76 | 0.755 |
84
+ | 0.4288 | 2.0568 | 10000 | 0.4527 | 0.761 | 0.859 | 0.742 | 0.873 | 0.802 | 0.75 | 0.761 | 0.756 |
85
+ | 0.4124 | 2.1596 | 10500 | 0.4477 | 0.762 | 0.861 | 0.736 | 0.891 | 0.806 | 0.749 | 0.762 | 0.756 |
86
+ | 0.4181 | 2.2624 | 11000 | 0.4569 | 0.759 | 0.857 | 0.741 | 0.87 | 0.8 | 0.748 | 0.759 | 0.754 |
87
+ | 0.4178 | 2.3653 | 11500 | 0.4469 | 0.762 | 0.861 | 0.741 | 0.879 | 0.804 | 0.751 | 0.762 | 0.757 |
88
+ | 0.4127 | 2.4681 | 12000 | 0.4448 | 0.764 | 0.863 | 0.742 | 0.881 | 0.806 | 0.753 | 0.764 | 0.759 |
89
+ | 0.419 | 2.5710 | 12500 | 0.4454 | 0.764 | 0.864 | 0.734 | 0.902 | 0.809 | 0.75 | 0.764 | 0.757 |
90
+ | 0.4232 | 2.6738 | 13000 | 0.4394 | 0.766 | 0.864 | 0.747 | 0.873 | 0.805 | 0.756 | 0.766 | 0.761 |
91
+ | 0.4226 | 2.7766 | 13500 | 0.4404 | 0.766 | 0.864 | 0.747 | 0.873 | 0.805 | 0.756 | 0.766 | 0.761 |
92
+ | 0.4196 | 2.8795 | 14000 | 0.4477 | 0.765 | 0.862 | 0.758 | 0.846 | 0.8 | 0.757 | 0.765 | 0.762 |
93
+ | 0.408 | 2.9823 | 14500 | 0.4497 | 0.763 | 0.862 | 0.745 | 0.871 | 0.803 | 0.752 | 0.763 | 0.758 |
94
+ | 0.4054 | 3.0852 | 15000 | 0.4404 | 0.765 | 0.865 | 0.749 | 0.865 | 0.803 | 0.755 | 0.765 | 0.76 |
95
+ | 0.4155 | 3.1880 | 15500 | 0.4466 | 0.764 | 0.863 | 0.74 | 0.885 | 0.806 | 0.752 | 0.764 | 0.758 |
96
+ | 0.4155 | 3.2908 | 16000 | 0.4417 | 0.765 | 0.864 | 0.744 | 0.879 | 0.806 | 0.754 | 0.765 | 0.76 |
97
+ | 0.4104 | 3.3937 | 16500 | 0.4407 | 0.766 | 0.866 | 0.742 | 0.887 | 0.808 | 0.755 | 0.766 | 0.761 |
98
+ | 0.4081 | 3.4965 | 17000 | 0.4406 | 0.765 | 0.866 | 0.753 | 0.859 | 0.802 | 0.756 | 0.765 | 0.762 |
99
+ | 0.4046 | 3.5993 | 17500 | 0.4384 | 0.768 | 0.868 | 0.742 | 0.891 | 0.81 | 0.756 | 0.768 | 0.762 |
100
+ | 0.4065 | 3.7022 | 18000 | 0.4443 | 0.766 | 0.866 | 0.742 | 0.888 | 0.808 | 0.754 | 0.766 | 0.76 |
101
+ | 0.4028 | 3.8050 | 18500 | 0.4438 | 0.768 | 0.866 | 0.746 | 0.882 | 0.808 | 0.757 | 0.768 | 0.763 |
102
+ | 0.4035 | 3.9079 | 19000 | 0.4453 | 0.766 | 0.865 | 0.753 | 0.862 | 0.804 | 0.758 | 0.766 | 0.763 |
103
+ | 0.4003 | 4.0107 | 19500 | 0.4426 | 0.767 | 0.865 | 0.75 | 0.87 | 0.806 | 0.757 | 0.767 | 0.763 |
104
+ | 0.4011 | 4.1135 | 20000 | 0.4423 | 0.767 | 0.867 | 0.741 | 0.892 | 0.81 | 0.755 | 0.767 | 0.761 |
105
+ | 0.3924 | 4.2164 | 20500 | 0.4394 | 0.768 | 0.867 | 0.75 | 0.874 | 0.807 | 0.758 | 0.768 | 0.764 |
106
+ | 0.4043 | 4.3192 | 21000 | 0.4421 | 0.768 | 0.867 | 0.744 | 0.887 | 0.809 | 0.757 | 0.768 | 0.762 |
107
+ | 0.395 | 4.4220 | 21500 | 0.4450 | 0.767 | 0.865 | 0.75 | 0.87 | 0.805 | 0.757 | 0.767 | 0.763 |
108
+ | 0.4072 | 4.5249 | 22000 | 0.4392 | 0.769 | 0.867 | 0.751 | 0.874 | 0.808 | 0.759 | 0.769 | 0.765 |
109
+ | 0.4055 | 4.6277 | 22500 | 0.4439 | 0.768 | 0.866 | 0.745 | 0.885 | 0.809 | 0.757 | 0.768 | 0.762 |
110
+ | 0.3987 | 4.7306 | 23000 | 0.4435 | 0.768 | 0.866 | 0.747 | 0.881 | 0.808 | 0.758 | 0.768 | 0.763 |
111
+ | 0.396 | 4.8334 | 23500 | 0.4430 | 0.768 | 0.867 | 0.745 | 0.885 | 0.809 | 0.757 | 0.768 | 0.763 |
112
+ | 0.4017 | 4.9362 | 24000 | 0.4421 | 0.769 | 0.867 | 0.745 | 0.887 | 0.81 | 0.757 | 0.769 | 0.763 |
113
+
114
+
115
+ ### Framework versions
116
+
117
+ - Transformers 4.53.1
118
+ - Pytorch 2.6.0+cu124
119
+ - Datasets 2.14.4
120
+ - Tokenizers 0.21.2
config.json ADDED
@@ -0,0 +1,33 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "architectures": [
3
+ "BertForSequenceClassification"
4
+ ],
5
+ "attention_probs_dropout_prob": 0.1,
6
+ "classifier_dropout": null,
7
+ "hidden_act": "gelu",
8
+ "hidden_dropout_prob": 0.1,
9
+ "hidden_size": 768,
10
+ "id2label": {
11
+ "0": "fake",
12
+ "1": "real"
13
+ },
14
+ "initializer_range": 0.02,
15
+ "intermediate_size": 3072,
16
+ "label2id": {
17
+ "fake": 0,
18
+ "real": 1
19
+ },
20
+ "layer_norm_eps": 1e-12,
21
+ "max_position_embeddings": 512,
22
+ "model_type": "bert",
23
+ "num_attention_heads": 12,
24
+ "num_hidden_layers": 12,
25
+ "pad_token_id": 0,
26
+ "position_embedding_type": "absolute",
27
+ "problem_type": "single_label_classification",
28
+ "torch_dtype": "float32",
29
+ "transformers_version": "4.53.1",
30
+ "type_vocab_size": 2,
31
+ "use_cache": true,
32
+ "vocab_size": 28996
33
+ }
model.safetensors ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c2c50b70f9717f2fa6cc531cbc0b711317ad4e99632ddb4df37ca89cde4d7f3b
3
+ size 433270768
runs/Jul14_10-21-56_8141a392dcea/events.out.tfevents.1752488542.8141a392dcea.5512.0 ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e5a22a085118d9c2811e3f0d7e05599a2584741808293f579b3a00ce8868e510
3
+ size 48290
special_tokens_map.json ADDED
@@ -0,0 +1,7 @@
 
 
 
 
 
 
 
 
1
+ {
2
+ "cls_token": "[CLS]",
3
+ "mask_token": "[MASK]",
4
+ "pad_token": "[PAD]",
5
+ "sep_token": "[SEP]",
6
+ "unk_token": "[UNK]"
7
+ }
tokenizer.json ADDED
The diff for this file is too large to render. See raw diff
 
tokenizer_config.json ADDED
@@ -0,0 +1,58 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ {
2
+ "added_tokens_decoder": {
3
+ "0": {
4
+ "content": "[PAD]",
5
+ "lstrip": false,
6
+ "normalized": false,
7
+ "rstrip": false,
8
+ "single_word": false,
9
+ "special": true
10
+ },
11
+ "100": {
12
+ "content": "[UNK]",
13
+ "lstrip": false,
14
+ "normalized": false,
15
+ "rstrip": false,
16
+ "single_word": false,
17
+ "special": true
18
+ },
19
+ "101": {
20
+ "content": "[CLS]",
21
+ "lstrip": false,
22
+ "normalized": false,
23
+ "rstrip": false,
24
+ "single_word": false,
25
+ "special": true
26
+ },
27
+ "102": {
28
+ "content": "[SEP]",
29
+ "lstrip": false,
30
+ "normalized": false,
31
+ "rstrip": false,
32
+ "single_word": false,
33
+ "special": true
34
+ },
35
+ "103": {
36
+ "content": "[MASK]",
37
+ "lstrip": false,
38
+ "normalized": false,
39
+ "rstrip": false,
40
+ "single_word": false,
41
+ "special": true
42
+ }
43
+ },
44
+ "clean_up_tokenization_spaces": true,
45
+ "cls_token": "[CLS]",
46
+ "do_basic_tokenize": true,
47
+ "do_lower_case": true,
48
+ "extra_special_tokens": {},
49
+ "mask_token": "[MASK]",
50
+ "model_max_length": 1000000000000000019884624838656,
51
+ "never_split": null,
52
+ "pad_token": "[PAD]",
53
+ "sep_token": "[SEP]",
54
+ "strip_accents": null,
55
+ "tokenize_chinese_chars": true,
56
+ "tokenizer_class": "BertTokenizer",
57
+ "unk_token": "[UNK]"
58
+ }
training_args.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:c25413d499c775d41a49c798775d6800a3963be7d4472825320d3c836e749a16
3
+ size 5368
vocab.txt ADDED
The diff for this file is too large to render. See raw diff