willtensora commited on
Commit
0617cab
·
verified ·
1 Parent(s): 35ecdc7

End of training

Browse files
Files changed (3) hide show
  1. README.md +135 -0
  2. adapter_model.bin +3 -0
  3. adapter_model.safetensors +1 -1
README.md ADDED
@@ -0,0 +1,135 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ library_name: peft
3
+ license: apache-2.0
4
+ base_model: EleutherAI/pythia-410m-deduped
5
+ tags:
6
+ - axolotl
7
+ - generated_from_trainer
8
+ model-index:
9
+ - name: bf58c716-f1ab-4447-897b-55ade88ff479
10
+ results: []
11
+ ---
12
+
13
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
14
+ should probably proofread and complete it, then remove this comment. -->
15
+
16
+ [<img src="https://raw.githubusercontent.com/axolotl-ai-cloud/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/axolotl-ai-cloud/axolotl)
17
+ <details><summary>See axolotl config</summary>
18
+
19
+ axolotl version: `0.4.1`
20
+ ```yaml
21
+ adapter: lora
22
+ base_model: EleutherAI/pythia-410m-deduped
23
+ batch_size: 8
24
+ bf16: true
25
+ chat_template: tokenizer_default_fallback_alpaca
26
+ datasets:
27
+ - data_files:
28
+ - 7b17ba91d0356ec9_train_data.json
29
+ ds_type: json
30
+ format: custom
31
+ path: /workspace/input_data/7b17ba91d0356ec9_train_data.json
32
+ type:
33
+ field_instruction: prompt
34
+ field_output: completion
35
+ format: '{instruction}'
36
+ no_input_format: '{instruction}'
37
+ system_format: '{system}'
38
+ system_prompt: ''
39
+ early_stopping_patience: 3
40
+ eval_steps: 50
41
+ flash_attention: true
42
+ gpu_memory_limit: 80GiB
43
+ gradient_checkpointing: true
44
+ group_by_length: true
45
+ hub_model_id: willtensora/bf58c716-f1ab-4447-897b-55ade88ff479
46
+ hub_strategy: checkpoint
47
+ learning_rate: 0.0002
48
+ logging_steps: 10
49
+ lora_alpha: 256
50
+ lora_dropout: 0.1
51
+ lora_r: 128
52
+ lora_target_linear: true
53
+ lr_scheduler: cosine
54
+ micro_batch_size: 1
55
+ model_type: AutoModelForCausalLM
56
+ num_epochs: 100
57
+ optimizer: adamw_bnb_8bit
58
+ output_dir: miner_id_24
59
+ pad_to_sequence_len: true
60
+ resize_token_embeddings_to_32x: false
61
+ sample_packing: false
62
+ save_steps: 50
63
+ sequence_len: 2048
64
+ special_tokens:
65
+ pad_token: <|endoftext|>
66
+ tokenizer_type: GPTNeoXTokenizerFast
67
+ train_on_inputs: false
68
+ trust_remote_code: true
69
+ val_set_size: 0.1
70
+ wandb_entity: ''
71
+ wandb_mode: online
72
+ wandb_project: Gradients-On-Demand
73
+ wandb_run: your_name
74
+ wandb_runid: default
75
+ warmup_ratio: 0.05
76
+ xformers_attention: true
77
+
78
+ ```
79
+
80
+ </details><br>
81
+
82
+ # bf58c716-f1ab-4447-897b-55ade88ff479
83
+
84
+ This model is a fine-tuned version of [EleutherAI/pythia-410m-deduped](https://huggingface.co/EleutherAI/pythia-410m-deduped) on the None dataset.
85
+ It achieves the following results on the evaluation set:
86
+ - Loss: 2.4376
87
+
88
+ ## Model description
89
+
90
+ More information needed
91
+
92
+ ## Intended uses & limitations
93
+
94
+ More information needed
95
+
96
+ ## Training and evaluation data
97
+
98
+ More information needed
99
+
100
+ ## Training procedure
101
+
102
+ ### Training hyperparameters
103
+
104
+ The following hyperparameters were used during training:
105
+ - learning_rate: 0.0002
106
+ - train_batch_size: 1
107
+ - eval_batch_size: 1
108
+ - seed: 42
109
+ - distributed_type: multi-GPU
110
+ - num_devices: 8
111
+ - total_train_batch_size: 8
112
+ - total_eval_batch_size: 8
113
+ - optimizer: Use OptimizerNames.ADAMW_BNB with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
114
+ - lr_scheduler_type: cosine
115
+ - lr_scheduler_warmup_steps: 153
116
+ - num_epochs: 100
117
+
118
+ ### Training results
119
+
120
+ | Training Loss | Epoch | Step | Validation Loss |
121
+ |:-------------:|:------:|:----:|:---------------:|
122
+ | No log | 0.0041 | 1 | 1.8043 |
123
+ | 1.4581 | 0.2033 | 50 | 1.4781 |
124
+ | 1.7801 | 0.4065 | 100 | 1.7743 |
125
+ | 1.6691 | 0.6098 | 150 | 1.7577 |
126
+ | 2.3973 | 0.8130 | 200 | 2.4376 |
127
+
128
+
129
+ ### Framework versions
130
+
131
+ - PEFT 0.13.2
132
+ - Transformers 4.46.0
133
+ - Pytorch 2.5.0+cu124
134
+ - Datasets 3.0.1
135
+ - Tokenizers 0.20.1
adapter_model.bin ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:8658270420351791da5bc7bdd099a5475ba1c88333639a0be7533fc90d191c94
3
+ size 201397322
adapter_model.safetensors CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:73a5f213038caea6d0f765ed163ac956e07f23817953a206db64adff8dc1c82e
3
  size 201353888
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:b6e1f70dc601243515fff2c879dffdd75dc3255311d25b53c0b3861ef4d158c8
3
  size 201353888