Update README.md
Browse files
README.md
CHANGED
|
@@ -3,3 +3,13 @@ license: other
|
|
| 3 |
license_name: yi-license
|
| 4 |
license_link: LICENSE
|
| 5 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 3 |
license_name: yi-license
|
| 4 |
license_link: LICENSE
|
| 5 |
---
|
| 6 |
+
Another AEZAKMI v2 finetune over Yi-34B-200K-rawrr-r3. Sequence length 2200
|
| 7 |
+
I was able to squeeze that in using Unsloth, script I used is in this repo.
|
| 8 |
+
Training took around 18 hours on local RTX 3090 Ti.
|
| 9 |
+
Will be uploading fp16 and exl2 soon. So far it seems like de-contaminating Yi worked nicely.
|
| 10 |
+
This lora goes over Yi-34B-200K-rawrr1-LORA-DPO-experimental-r3 lora.
|
| 11 |
+
So first get Yi-34B-200K llamafied, merge in Yi-34B-200K-rawrr1-LORA-DPO-experimental-r3, then merge in this lora.
|
| 12 |
+
|
| 13 |
+
Credits for mlabonne (I was using his Mistral fine-tuning script pieces for dataset preparation), Daniel Han and Michael Han (Unsloth AI team)
|
| 14 |
+
|
| 15 |
+
[<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" alt="made with Unsloth" width="400" height="64"/>](https://github.com/unslothai/unsloth)
|