Hugging Face's logo

shenxq
/
zephyr-7b-dpo-qlora

PEFT
TensorBoard
Safetensors
mistral
alignment-handbook
Generated from Trainer
trl
dpo
4-bit precision
bitsandbytes
Model card Files Metrics Community
shenxq commited on
Commit
29be7a4
·
verified ·
1 Parent(s): d81199e

Model save

Browse files
Files changed (8) hide show
  1. README.md +22 -26
  2. adapter_config.json +4 -4
  3. adapter_model.safetensors +2 -2
  4. all_results.json +16 -16
  5. eval_results.json +12 -12
  6. train_results.json +4 -4
  7. trainer_state.json +0 -0
  8. training_args.bin +1 -1
README.md CHANGED
@@ -2,13 +2,9 @@
2
  license: apache-2.0
3
  library_name: peft
4
  tags:
5
- - alignment-handbook
6
- - generated_from_trainer
7
  - trl
8
  - dpo
9
  - generated_from_trainer
10
- datasets:
11
- - snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset
12
  base_model: mistralai/Mistral-7B-v0.1
13
  model-index:
14
  - name: zephyr-7b-dpo-qlora
@@ -20,17 +16,17 @@ should probably proofread and complete it, then remove this comment. -->
20
 
21
  # zephyr-7b-dpo-qlora
22
 
23
- This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the snorkelai/Snorkel-Mistral-PairRM-DPO-Dataset dataset.
24
  It achieves the following results on the evaluation set:
25
- - Loss: 0.6707
26
- - Rewards/chosen: -0.2860
27
- - Rewards/rejected: -0.3548
28
- - Rewards/accuracies: 0.5983
29
- - Rewards/margins: 0.0687
30
- - Logps/rejected: -367.6676
31
- - Logps/chosen: -351.0971
32
- - Logits/rejected: -2.5801
33
- - Logits/chosen: -2.5726
34
 
35
  ## Model description
36
 
@@ -65,18 +61,18 @@ The following hyperparameters were used during training:
65
 
66
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
67
  |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
68
- | 0.6932 | 0.08 | 100 | 0.6930 | -0.0030 | -0.0033 | 0.5220 | 0.0003 | -332.5208 | -322.7949 | -2.4978 | -2.4908 |
69
- | 0.6921 | 0.16 | 200 | 0.6927 | -0.0232 | -0.0243 | 0.5183 | 0.0011 | -334.6197 | -324.8167 | -2.4970 | -2.4900 |
70
- | 0.6913 | 0.24 | 300 | 0.6919 | -0.0414 | -0.0441 | 0.5340 | 0.0027 | -336.6059 | -326.6393 | -2.4967 | -2.4895 |
71
- | 0.6893 | 0.32 | 400 | 0.6891 | -0.0791 | -0.0883 | 0.5547 | 0.0093 | -341.0244 | -330.4017 | -2.5023 | -2.4953 |
72
- | 0.6724 | 0.4 | 500 | 0.6844 | -0.2018 | -0.2253 | 0.5530 | 0.0235 | -354.7256 | -342.6785 | -2.5100 | -2.5029 |
73
- | 0.6849 | 0.48 | 600 | 0.6805 | -0.3366 | -0.3770 | 0.5597 | 0.0404 | -369.8958 | -356.1591 | -2.5412 | -2.5347 |
74
- | 0.6503 | 0.56 | 700 | 0.6774 | -0.4376 | -0.4919 | 0.5630 | 0.0543 | -381.3843 | -366.2523 | -2.5492 | -2.5431 |
75
- | 0.6841 | 0.64 | 800 | 0.6735 | -0.3183 | -0.3788 | 0.5913 | 0.0605 | -370.0676 | -354.3206 | -2.5662 | -2.5592 |
76
- | 0.6773 | 0.72 | 900 | 0.6724 | -0.3986 | -0.4678 | 0.5887 | 0.0692 | -378.9693 | -362.3546 | -2.5774 | -2.5706 |
77
- | 0.657 | 0.8 | 1000 | 0.6711 | -0.2774 | -0.3440 | 0.5997 | 0.0666 | -366.5909 | -350.2372 | -2.5784 | -2.5708 |
78
- | 0.6577 | 0.88 | 1100 | 0.6706 | -0.2934 | -0.3628 | 0.5993 | 0.0693 | -368.4680 | -351.8376 | -2.5805 | -2.5729 |
79
- | 0.6444 | 0.96 | 1200 | 0.6708 | -0.2860 | -0.3547 | 0.5993 | 0.0687 | -367.6592 | -351.0949 | -2.5801 | -2.5725 |
80
 
81
 
82
  ### Framework versions
 
2
  license: apache-2.0
3
  library_name: peft
4
  tags:
 
 
5
  - trl
6
  - dpo
7
  - generated_from_trainer
 
 
8
  base_model: mistralai/Mistral-7B-v0.1
9
  model-index:
10
  - name: zephyr-7b-dpo-qlora
 
16
 
17
  # zephyr-7b-dpo-qlora
18
 
19
+ This model is a fine-tuned version of [mistralai/Mistral-7B-v0.1](https://huggingface.co/mistralai/Mistral-7B-v0.1) on the None dataset.
20
  It achieves the following results on the evaluation set:
21
+ - Loss: 0.6668
22
+ - Rewards/chosen: -0.2672
23
+ - Rewards/rejected: -0.3491
24
+ - Rewards/accuracies: 0.6137
25
+ - Rewards/margins: 0.0819
26
+ - Logps/rejected: -378.9569
27
+ - Logps/chosen: -361.0521
28
+ - Logits/rejected: -2.5949
29
+ - Logits/chosen: -2.5884
30
 
31
  ## Model description
32
 
 
61
 
62
  | Training Loss | Epoch | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
63
  |:-------------:|:-----:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
64
+ | 0.6933 | 0.08 | 100 | 0.6930 | -0.0077 | -0.0080 | 0.5177 | 0.0004 | -344.8478 | -335.0984 | -2.4838 | -2.4768 |
65
+ | 0.6926 | 0.16 | 200 | 0.6923 | -0.0138 | -0.0155 | 0.5427 | 0.0017 | -345.5920 | -335.7114 | -2.4836 | -2.4766 |
66
+ | 0.6906 | 0.24 | 300 | 0.6917 | -0.0130 | -0.0161 | 0.5523 | 0.0031 | -345.6560 | -335.6324 | -2.4879 | -2.4809 |
67
+ | 0.6884 | 0.32 | 400 | 0.6898 | -0.0075 | -0.0146 | 0.5807 | 0.0071 | -345.4990 | -335.0794 | -2.4972 | -2.4901 |
68
+ | 0.6753 | 0.4 | 500 | 0.6856 | -0.1385 | -0.1579 | 0.5630 | 0.0194 | -359.8317 | -348.1783 | -2.4986 | -2.4916 |
69
+ | 0.6839 | 0.48 | 600 | 0.6815 | -0.3188 | -0.3556 | 0.5667 | 0.0368 | -379.6049 | -366.2155 | -2.5394 | -2.5333 |
70
+ | 0.6535 | 0.56 | 700 | 0.6770 | -0.4204 | -0.4741 | 0.5763 | 0.0537 | -391.4496 | -376.3719 | -2.5483 | -2.5425 |
71
+ | 0.6764 | 0.64 | 800 | 0.6724 | -0.2481 | -0.3087 | 0.5990 | 0.0606 | -374.9128 | -359.1413 | -2.5714 | -2.5651 |
72
+ | 0.6753 | 0.72 | 900 | 0.6704 | -0.4283 | -0.5062 | 0.5983 | 0.0780 | -394.6671 | -377.1592 | -2.5807 | -2.5750 |
73
+ | 0.6459 | 0.8 | 1000 | 0.6680 | -0.2406 | -0.3163 | 0.6127 | 0.0757 | -375.6733 | -358.3894 | -2.5924 | -2.5858 |
74
+ | 0.6541 | 0.88 | 1100 | 0.6670 | -0.2806 | -0.3625 | 0.6157 | 0.0820 | -380.2968 | -362.3882 | -2.5942 | -2.5878 |
75
+ | 0.6422 | 0.96 | 1200 | 0.6669 | -0.2657 | -0.3473 | 0.6157 | 0.0817 | -378.7738 | -360.8972 | -2.5963 | -2.5898 |
76
 
77
 
78
  ### Framework versions
adapter_config.json CHANGED
@@ -19,12 +19,12 @@
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
22
- "gate_proj",
23
- "k_proj",
24
  "v_proj",
25
- "q_proj",
26
- "down_proj",
27
  "o_proj",
 
 
 
28
  "up_proj"
29
  ],
30
  "task_type": "CAUSAL_LM"
 
19
  "rank_pattern": {},
20
  "revision": null,
21
  "target_modules": [
 
 
22
  "v_proj",
23
+ "k_proj",
 
24
  "o_proj",
25
+ "gate_proj",
26
+ "down_proj",
27
+ "q_proj",
28
  "up_proj"
29
  ],
30
  "task_type": "CAUSAL_LM"
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:aa34178bd80f8d3a5f7293547ed95d1e9a509a94b2df87e75b782f3843a5f42c
3
- size 83945744
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:6052991d04f63299b92ad5b7794163db85499e588fc3031f814a9877029dd414
3
+ size 167832240
all_results.json CHANGED
@@ -1,21 +1,21 @@
1
  {
2
  "epoch": 1.0,
3
- "eval_logits/chosen": -2.5725526809692383,
4
- "eval_logits/rejected": -2.5801451206207275,
5
- "eval_logps/chosen": -351.0971374511719,
6
- "eval_logps/rejected": -367.6675720214844,
7
- "eval_loss": 0.6707386374473572,
8
- "eval_rewards/accuracies": 0.5983333587646484,
9
- "eval_rewards/chosen": -0.28601789474487305,
10
- "eval_rewards/margins": 0.0687330961227417,
11
- "eval_rewards/rejected": -0.35475102066993713,
12
- "eval_runtime": 1581.2259,
13
  "eval_samples": 2994,
14
- "eval_samples_per_second": 1.893,
15
- "eval_steps_per_second": 0.237,
16
- "train_loss": 0.67484232365751,
17
- "train_runtime": 40604.4701,
18
  "train_samples": 19996,
19
- "train_samples_per_second": 0.492,
20
- "train_steps_per_second": 0.031
21
  }
 
1
  {
2
  "epoch": 1.0,
3
+ "eval_logits/chosen": -2.5884382724761963,
4
+ "eval_logits/rejected": -2.5948920249938965,
5
+ "eval_logps/chosen": -361.0521240234375,
6
+ "eval_logps/rejected": -378.9569091796875,
7
+ "eval_loss": 0.6668171882629395,
8
+ "eval_rewards/accuracies": 0.6136666536331177,
9
+ "eval_rewards/chosen": -0.26719996333122253,
10
+ "eval_rewards/margins": 0.08193866163492203,
11
+ "eval_rewards/rejected": -0.3491385877132416,
12
+ "eval_runtime": 1605.8768,
13
  "eval_samples": 2994,
14
+ "eval_samples_per_second": 1.864,
15
+ "eval_steps_per_second": 0.234,
16
+ "train_loss": 0.6736802890646257,
17
+ "train_runtime": 42164.1735,
18
  "train_samples": 19996,
19
+ "train_samples_per_second": 0.474,
20
+ "train_steps_per_second": 0.03
21
  }
eval_results.json CHANGED
@@ -1,16 +1,16 @@
1
  {
2
  "epoch": 1.0,
3
- "eval_logits/chosen": -2.5725526809692383,
4
- "eval_logits/rejected": -2.5801451206207275,
5
- "eval_logps/chosen": -351.0971374511719,
6
- "eval_logps/rejected": -367.6675720214844,
7
- "eval_loss": 0.6707386374473572,
8
- "eval_rewards/accuracies": 0.5983333587646484,
9
- "eval_rewards/chosen": -0.28601789474487305,
10
- "eval_rewards/margins": 0.0687330961227417,
11
- "eval_rewards/rejected": -0.35475102066993713,
12
- "eval_runtime": 1581.2259,
13
  "eval_samples": 2994,
14
- "eval_samples_per_second": 1.893,
15
- "eval_steps_per_second": 0.237
16
  }
 
1
  {
2
  "epoch": 1.0,
3
+ "eval_logits/chosen": -2.5884382724761963,
4
+ "eval_logits/rejected": -2.5948920249938965,
5
+ "eval_logps/chosen": -361.0521240234375,
6
+ "eval_logps/rejected": -378.9569091796875,
7
+ "eval_loss": 0.6668171882629395,
8
+ "eval_rewards/accuracies": 0.6136666536331177,
9
+ "eval_rewards/chosen": -0.26719996333122253,
10
+ "eval_rewards/margins": 0.08193866163492203,
11
+ "eval_rewards/rejected": -0.3491385877132416,
12
+ "eval_runtime": 1605.8768,
13
  "eval_samples": 2994,
14
+ "eval_samples_per_second": 1.864,
15
+ "eval_steps_per_second": 0.234
16
  }
train_results.json CHANGED
@@ -1,8 +1,8 @@
1
  {
2
  "epoch": 1.0,
3
- "train_loss": 0.67484232365751,
4
- "train_runtime": 40604.4701,
5
  "train_samples": 19996,
6
- "train_samples_per_second": 0.492,
7
- "train_steps_per_second": 0.031
8
  }
 
1
  {
2
  "epoch": 1.0,
3
+ "train_loss": 0.6736802890646257,
4
+ "train_runtime": 42164.1735,
5
  "train_samples": 19996,
6
+ "train_samples_per_second": 0.474,
7
+ "train_steps_per_second": 0.03
8
  }
trainer_state.json CHANGED
The diff for this file is too large to render. See raw diff
 
training_args.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:387786aa767b34abf9b422cc0cdb3b0676810f6854c743acf6d68e01b38091c3
3
  size 5176
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:fee41ec1b45b86b78f6aa2bc1bd14d3332831518f258d69fa9acb1393bd5b3ed
3
  size 5176