File size: 3,785 Bytes
fdf5890 0aab70f fdf5890 ce836fb 0aab70f 5817a52 0aab70f fdf5890 0aab70f 5817a52 0aab70f 5817a52 0aab70f 5817a52 0aab70f 5817a52 0aab70f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 |
---
tags:
- chat
- roleplay
- storywriting
- qwen3
- finetune
Language:
- En
Pipeline_tag: text-generation
Tags:
- Chat
base_model:
- Qwen/Qwen3-235B-A22B
---
[English](./non-lore-README.md) | [Francais](./French-README.md)
It's an SFT ontop of the largest Qwen which nobody seems to have done yet, Trained with a collection of normal Austral(Books, RP Logs, LNs, etc) datasets. I do not totally endorse the model yet and i think there's much work to be done in trying to make a decensored and well-writing finetune of this model but I just released this to give everyone a slight taste of a qwen3 finetune.
It was also a way for us to test out some Optims to actually get this model to train, Thanks to Intervitens <3
We used torchtune & an experimental hacky pytorch build: https://github.com/pytorch/pytorch/pull/156203
We trained this model over 24 Hours on 8xB200s. Graciously provided by Deepinfra & Cognitive Computations.
Speeds were similar to a 70B trained with roughly the same data.
## Prompting
Model has been tuned with the ChatML formatting. A typical input would look like this:
```py
<|im_start|>system
system-prompt<|im_end|>
<|im_start|>user
user-prompt<|im_end|>
<|im_start|>assistant
<think>
</think>
assistant-prompt<|im_end|>
```
## Torchtune config
Thank you so much for Intervitens for helping train this model:
<details><summary>See Torchtune Trainer config</summary>
```yaml
output_dir: ./qwen3_235B_A22B_austral/full
tokenizer:
_component_: torchtune.models.qwen3.qwen3_tokenizer
path: ./Qwen3-235B-A22B-tt/vocab.json
merges_file: ./Qwen3-235B-A22B-tt/merges.txt
max_seq_len: 32768
dataset:
_component_: torchtune.datasets.pretokenized_dataset
source: IntervitensInc/test_235B_2-pack
split: train
packed: true
seed: 42
shuffle: false
model:
_component_: torchtune.models.qwen3.qwen3_moe_235b_a22b
checkpointer:
_component_: torchtune.training.FullModelTorchTuneCheckpointer
checkpoint_dir: ./Qwen3-235B-A22B-tt
checkpoint_files:
- model-00001-of-00001.bin
recipe_checkpoint: null
output_dir: ${output_dir}
model_type: QWEN3_MOE
resume_from_checkpoint: false
enable_async_checkpointing: false
batch_size: 1
epochs: 4
optimizer:
_component_: torchao.optim.AdamW8bit
lr: 3.0e-06
lr_scheduler:
_component_: torchtune.training.lr_schedulers.get_rex_scheduler
num_warmup_steps: 100
loss:
_component_: torchtune.modules.loss.LinearCrossEntropyLoss
max_steps_per_epoch: null
gradient_accumulation_steps: 1
clip_grad_norm: null
compile:
model: true
loss: true
scale_grads: true
optimizer_step: false
optimizer_in_bwd: true
device: cuda
enable_activation_checkpointing: true
enable_activation_offloading: true
custom_sharded_layers:
- tok_embeddings
- output
fsdp_cpu_offload: false
dtype: bf16
metric_logger:
_component_: torchtune.training.metric_logging.WandBLogger
project: qwen3-235-a22b-austral
log_every_n_steps: 1
log_peak_memory_stats: true
log_level: INFO
```
</details><br>
## Credits
Thank you to [Lucy Knada](https://huggingface.co/lucyknada), [Auri](https://huggingface.co/Auri), [Intervitens](https://huggingface.co/intervitens), [Deepinfra](https://deepinfra.com/), [Cognitive Computations](https://huggingface.co/cognitivecomputations) and the rest of [Anthracite](https://huggingface.co/anthracite-org) &
## Training
The training was done for 4 epochs. We used 8 x [B200s](https://www.nvidia.com/en-us/data-center/dgx-b200/) GPUs graciously provided by [Deepinfra](https://deepinfra.com/) for the full-parameter fine-tuning of the model, Tuning was done all thanks to Intervitens.
## Safety
It's still aligned to the beliefs of the Chinese Communist Party:
 |