jukofyork
/

command-a-03-2025-uncut

Text Generation

Model card Files Files and versions

jukofyork commited on Sep 20

Commit

145d5ab

·

verified ·

1 Parent(s): 939ef66

Update README.md

Files changed (1) hide show

README.md +3 -1

README.md CHANGED Viewed

@@ -120,7 +120,7 @@ using ~200M tokens (ie: ~100M positive and ~100M negative) from:
 - [jukofyork/instruction-responses-500MB](https://huggingface.co/datasets/jukofyork/instruction-responses-500MB)
 - [jukofyork/instruction-refusals-500MB](https://huggingface.co/datasets/jukofyork/instruction-refusals-500MB)
-taking just under 4 days:
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/bOMISzLsmjimXDZ2k72Mu.png)
@@ -128,6 +128,8 @@ taking just under 4 days:
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/7BJJK6UKRDCgqEqEIwlkA.png)
 ---
 The control adapter was then converted to a LoRA using [control_adapter_to_lora.py](https://github.com/jukofyork/qlora-pipe-lite/blob/main/control_adapter_to_lora.py):

 - [jukofyork/instruction-responses-500MB](https://huggingface.co/datasets/jukofyork/instruction-responses-500MB)
 - [jukofyork/instruction-refusals-500MB](https://huggingface.co/datasets/jukofyork/instruction-refusals-500MB)
+taking just under 4 days using 6x `RTX A6000` over 3 machines:
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/bOMISzLsmjimXDZ2k72Mu.png)
 ![image/png](https://cdn-uploads.huggingface.co/production/uploads/65995c45539c808e84c38bf1/7BJJK6UKRDCgqEqEIwlkA.png)
+(hence the 30 batch size: `(num_gpus / pipeline_stages) * gradient_accumulation_steps = (6 / 2) * 10 = 30`)
 ---
 The control adapter was then converted to a LoRA using [control_adapter_to_lora.py](https://github.com/jukofyork/qlora-pipe-lite/blob/main/control_adapter_to_lora.py):