End of training

README.md

@@ -0,0 +1,80 @@
+---
+library_name: peft
+license: llama2
+base_model: meta-llama/Llama-2-7b-hf
+tags:
+- trl
+- dpo
+- generated_from_trainer
+model-index:
+- name: Llama-2-7b-hf-DPO-LookAhead-5_Q2_TTree1.4_TT0.9_TP0.7_TE0.2_V3
+  results: []
+---
+
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# Llama-2-7b-hf-DPO-LookAhead-5_Q2_TTree1.4_TT0.9_TP0.7_TE0.2_V3
+
+This model is a fine-tuned version of [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf) on the None dataset.
+It achieves the following results on the evaluation set:
+- Loss: 0.8622
+- Rewards/chosen: -1.8032
+- Rewards/rejected: -1.8934
+- Rewards/accuracies: 0.4167
+- Rewards/margins: 0.0902
+- Logps/rejected: -178.6097
+- Logps/chosen: -144.0242
+- Logits/rejected: -0.2567
+- Logits/chosen: -0.2341
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 5e-05
+- train_batch_size: 2
+- eval_batch_size: 2
+- seed: 42
+- gradient_accumulation_steps: 2
+- total_train_batch_size: 4
+- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
+- lr_scheduler_type: cosine
+- lr_scheduler_warmup_steps: 10
+- num_epochs: 3
+
+### Training results
+
+| Training Loss | Epoch  | Step | Validation Loss | Rewards/chosen | Rewards/rejected | Rewards/accuracies | Rewards/margins | Logps/rejected | Logps/chosen | Logits/rejected | Logits/chosen |
+|:-------------:|:------:|:----:|:---------------:|:--------------:|:----------------:|:------------------:|:---------------:|:--------------:|:------------:|:---------------:|:-------------:|
+| 0.7186        | 0.3012 | 75   | 0.6922          | 0.0088         | -0.0056          | 0.6667             | 0.0144          | -159.7317      | -125.9045    | 0.2763          | 0.3051        |
+| 0.6878        | 0.6024 | 150  | 0.6645          | 0.0065         | -0.0784          | 0.6667             | 0.0850          | -160.4602      | -125.9270    | 0.2430          | 0.2714        |
+| 0.7115        | 0.9036 | 225  | 0.6671          | 0.1245         | 0.0380           | 0.5833             | 0.0865          | -159.2964      | -124.7477    | 0.2585          | 0.2872        |
+| 0.2588        | 1.2048 | 300  | 0.5773          | -0.4124        | -0.9074          | 0.6667             | 0.4951          | -168.7503      | -130.1161    | 0.1854          | 0.2129        |
+| 0.5429        | 1.5060 | 375  | 0.6801          | -0.4887        | -0.7667          | 0.5                | 0.2780          | -167.3426      | -130.8791    | 0.0976          | 0.1239        |
+| 0.3313        | 1.8072 | 450  | 0.7539          | -0.6406        | -0.7950          | 0.5                | 0.1545          | -167.6264      | -132.3980    | 0.0143          | 0.0407        |
+| 0.2905        | 2.1084 | 525  | 0.8112          | -1.3875        | -1.4781          | 0.4167             | 0.0906          | -174.4566      | -139.8674    | -0.1544         | -0.1306       |
+| 0.1737        | 2.4096 | 600  | 0.8469          | -1.9078        | -2.0075          | 0.4167             | 0.0997          | -179.7509      | -145.0706    | -0.2506         | -0.2282       |
+| 0.2314        | 2.7108 | 675  | 0.8622          | -1.8032        | -1.8934          | 0.4167             | 0.0902          | -178.6097      | -144.0242    | -0.2567         | -0.2341       |
+
+
+### Framework versions
+
+- PEFT 0.12.0
+- Transformers 4.45.2
+- Pytorch 2.4.0+cu121
+- Datasets 3.2.0
+- Tokenizers 0.20.3