Initial commit

.gitattributes

@@ -0,0 +1,35 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text

LICENSE

@@ -0,0 +1,31 @@
+The following applies to all files in this repository, unless otherwise noted:
+
+Copyright (c) 2024 IPAI Aleph Alpha Research GmbH. All rights reserved.
+
+This project is licensed under the terms of the Open Aleph License 1.0, available at
+https://github.com/Aleph-Alpha/.github/blob/main/oal.pdf
+
+---
+Excerpt from the license text:
+
+Subject to the terms and conditions of this License, the Licensor grants you a non-exclusive, worldwide,
+non-transferable, non-sublicensable, and royalty-free limited right to use, copy, modify, distribute, make
+otherwise publicly available, and reproduce the Works and Derivative Works under Licensor’s copyright,
+for any Non-Commercial and Non-Administrative purpose.
+You may not use, copy, modify, distribute, make otherwise publicly available, reproduce, or sublicense the
+Works or Derivative Works except as expressly provided under and in accordance with this License.
+Your rights granted under this License will automatically terminate if you fail to comply with any of the
+terms of this License.
+
+EXCEPT FOR DAMAGES CAUSED BY INTENT OR FRAUDULENTLY CONCEALED
+DEFECTS, AND EXCEPT FOR DAMAGES RESULTING FROM BREACH OF ANY
+WARRANTY OR GUARANTEE EXPRESSLY GIVEN BY LICENSOR IN THE OPEN ALEPH LICENSE,
+IN NO EVENT WILL LICENSOR BE LIABLE TO YOU ON ANY LEGAL THEORY FOR ANY
+DAMAGES ARISING OUT OF THE OPEN ALEPH LICENSE OR THE USE OF THE WORK. ANY
+MANDATORY STATUTORY LIABILITY UNDER APPLICABLE LAW REMAINS
+UNAFFECTED.
+
+EXCEPT AS EXPRESSLY STATED IN THIS LICENSE OR REQUIRED BY APPLICABLE
+LAW, THE WORKS ARE PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES
+OF ANY KIND INCLUDING, WITHOUT LIMITATION, ANY WARRANTIES REGARDING
+THE CONTENTS, ACCURACY, OR FITNESS FOR A PARTICULAR PURPOSE.

README.md

@@ -0,0 +1,50 @@
+---
+license: other
+license_name: open-aleph-license
+license_link: LICENSE
+library_name: scaling
+pipeline_tag: text-generation
+---
+
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/671a0238b080a748c29b8fea/v1rfcKVaL8vnjuCqWUmI-.png)
+
+
+# u-μP: Stable training in low precision for a significant speed-up and memory reduction during training
+
+
+![image/png](https://cdn-uploads.huggingface.co/production/uploads/671a0238b080a748c29b8fea/F1-zbAXF5LGvxpIRrYfU4.png)
+
+
+This Repository holds the model weights for the 7B u-μP models trained at Aleph Alpha Research for 72k steps (300B tokens). Please note, that the released checkpoints are not fully converged models and are intended for research use only.
+
+You can find all model weights and their corresponding safetensors conversions at the following links:
+- [umup-research-7b-bf16](https://huggingface.co/Aleph-Alpha/umup-research-7b-bf16)
+- [umup-research-7b-fp8](https://huggingface.co/Aleph-Alpha/umup-research-7b-fp8)
+- [sp-baseline-research-7b-bf16](https://huggingface.co/Aleph-Alpha/sp-baseline-research-7b-bf16)
+
+The Maximal Update Parametrization (μP) aims to make the optimal hyperparameters (HPs) of a model-independent of its size, allowing them to be swept using a cheap proxy model rather than the full-size target model. We present a new scheme, u-μP, which improves upon μP by combining it with Unit Scaling, a method for designing models that makes them easy to train in low precision. The two techniques have a natural affinity: μP ensures that the scale of activations is independent of model size, and Unit Scaling ensures that activations, weights, and gradients begin training with a scale of one. This synthesis opens the door to a simpler scheme, whose default values are near-optimal. This in turn facilitates a more efficient sweeping strategy, with u-μP models reaching a lower loss than comparable μP models and working out-of-the-box in FP8.
+
+
+
+If you want to learn more details about u-μP, check out our [blog post](https://aleph-alpha.com/in-awe-at-the-scale-of-these-tensors-a-gentle-introduction-to-unit-scaled-maximal-update-parametrization/) and our [paper](https://arxiv.org/abs/2407.17465).
+
+Unit-Scaled Maximal Update Parametrization (u-μP) is available in [Scaling](https://github.com/Aleph-Alpha/scaling), our official large-scale training codebase. Please note, that FP8-trained checkpoints only work on chips with FP8 support, like the Hopper architecture. 
+
+# Usage
+You can generate tokens with the [Scaling](https://github.com/Aleph-Alpha/scaling) inference implementation:
+
+```python
+from scaling.transformer.inference import TransformerInferenceModule
+from pathlib import Path
+
+ckpt_path = Path("<path_to_repo>/7B_umup_fp8")
+
+model = TransformerInferenceModule.from_checkpoint(ckpt_path)
+
+prompt = "Once upon a time"
+
+output = model.generate(max_tokens=100, input_text=prompt)
+
+print(output.completion_text)
+```

config.yml

@@ -0,0 +1,101 @@
+optimizer:
+  allreduce_bucket_size: 500000000
+  beta1: 0.9
+  beta2: 0.95
+  debug_log: false
+  eps: 1e-08
+  gradient_clipping: 0.0
+  zero: true
+  zero_save_static: false
+topology:
+  activation_checkpointing_type: disabled
+  global_batch_size: 1024
+  gradient_accumulation_steps: 4
+  micro_batch_size: 2
+  model_parallel_size: 1
+  pipe_parallel_size: 2
+  pipe_partition_method: balanced
+  pipe_partition_overwrite: null
+  sequence_parallel: false
+trainer:
+  seed: 42
+  train_iterations: 72000
+training:
+  allow_missing_params_in_optimizer: true
+training_groups:
+- group_name: param_group
+  independent_weight_decay: true
+  learning_rate_scheduler:
+    learning_rate: 11.313708498984761
+    learning_rate_decay_iters: 72000
+    learning_rate_decay_style: cosine
+    learning_rate_minimum: 1.131370849898476
+    learning_rate_warmup_steps: 500
+  parameters_exclude:
+  - norm
+  weight_decay: 0.0001221
+transformer_architecture:
+  attention_bias: false
+  attention_num_kv_heads: null
+  attention_qkv_in_one: true
+  dropout_after_attention: 0.0
+  dropout_after_mlp: 0.0
+  dropout_attention_probs: 0.0
+  dropout_embedding: 0.0
+  dropout_image_encoder: 0.0
+  fp8_config_attention:
+    dtypes_forward:
+      left_dtype: e4m3
+      right_dtype: e4m3
+    dtypes_grad_input:
+      left_dtype: e5m2
+      right_dtype: e4m3
+    dtypes_grad_weight:
+      left_dtype: e4m3
+      right_dtype: e5m2
+  fp8_config_mlp:
+    dtypes_forward:
+      left_dtype: e4m3
+      right_dtype: e4m3
+    dtypes_grad_input:
+      left_dtype: e5m2
+      right_dtype: e4m3
+    dtypes_grad_weight:
+      left_dtype: e4m3
+      right_dtype: e5m2
+  hidden_size: 4096
+  image_encoder: false
+  key_query_norm: false
+  layernorm:
+    layernorm_epsilon: 1e-05
+    optimization_type: torch
+  local_attention_window_size: null
+  masked_softmax:
+    kernel: flash_attention
+    scale: 1.0
+    softmax_in_fp32: false
+  mlp_bias: false
+  mlp_factor: 2.66796875
+  mlp_type: swiglu
+  norm_type: rms
+  num_attention_heads: 32
+  num_layers: 32
+  num_local_attention_heads: 0
+  precision: bfloat16
+  relative_position_embedding_type: rotary_complex
+  reset_attention_mask: false
+  reset_position_ids: false
+  rotary_embedding_base: 10000
+  rotary_percentage: 1.0
+  sequence_length: 4096
+  umup:
+    act_mult: 1.0
+    attn_mult: 1.0
+    enable: true
+    loss_mult: 1.0
+    normalize_depth_to_num_layers: true
+    residual_attn_ratio: 0.25
+    residual_mult: 1.0
+  vocab_file: null
+  vocab_size: 65536
+  weight_tying: false

model_state_layer_0_EmbeddingInput.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4a231ae564fec7d0538547295cd6a650f82e1907da3c2cfdb0025ab38cb241a7
+size 536872395

model_state_layer_10_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7faa4cee189672fd4370b7a4567d95f428a6ba155812a53cb6435068f089744
+size 402803885

model_state_layer_11_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:33d381483ac987da4f1def484f8e7d838c8e8044d18c641c45d79aedd1a72854
+size 402803885

model_state_layer_12_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:076b074089a7dd7c700f306e94be5db9389642562937b91b6afd65bc3a0fe4dd
+size 402803885

model_state_layer_13_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6d30169d26deb717b0e443aeb85846e82e531ae959b8a2814b2bf7cf44c15ae3
+size 402803885

model_state_layer_14_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:917156f9fef24f5159fae1dad1defa3bc1bae042dc31d365fd39f50af0f463c6
+size 402803885

model_state_layer_15_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1c82364c46d7b3d67fac05474c46b0dc65abf836ee53e14c4e31fa4b17358995
+size 402803885

model_state_layer_16_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7f65efab7e7296718512f8f272f0c32ce0c06e1d31394e40e63e17fd9bf1db0
+size 402803885

model_state_layer_17_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:76cb8a177cf9a77c105bc424a4a3f6df112abb6f53c74e9a741826cb0183199e
+size 402803885

model_state_layer_18_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ec06f64d40378137477c2da30503aec73f6e1489a385476ef16d94c504917b9e
+size 402803885

model_state_layer_19_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4ff91b52269e4e124b52b10bf34ce15e8d23167e211a99813918dca1bf2b6112
+size 402803885

model_state_layer_1_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e14a6e1c964c8da62a343b7da2d45404f8b35f83b6afea7b987cb6d0c2a91b81
+size 402803874

model_state_layer_20_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3f7c72d971db74f628064286fb0f7ecf416e8ccb893c4074aeddb993d89d30bc
+size 402803885

model_state_layer_21_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:62dddec6d330e9f61406cab3a3fbc1570094f22714ed0cb3e11a9338726becd9
+size 402803885

model_state_layer_22_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79fb6dfeb425ac8b57b50214d8dc795ce6cc91b58ad357e4c715959eb995b148
+size 402803885

model_state_layer_23_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:19f399896711cf4f0b1e59ca75982102eb35720071c606920a64364ed985167b
+size 402803885

model_state_layer_24_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c54ef4d647ae58c490591d44974048349e4380b22ff9a9274fa64ad62ddf0a5d
+size 402803885

model_state_layer_25_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5395e624161287112d852bf445cccd45fe161ffdc9f39a943ee6be96dc8c6c5d
+size 402803885

model_state_layer_26_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:ad9cfc0b9f9210abc5f243c1e5b54614c6aa43dbdf9e314823295c86f16991ac
+size 402803885

model_state_layer_27_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b0d8704348468da90d998fe364fd797ad1bb8c241f057da77f51abb09ea89359
+size 402803885

model_state_layer_28_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:41b864c7ff06f2f86abf422b4dc2711a684f513920771382a6aff6f8ef2e3bb2
+size 402803885

model_state_layer_29_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:491bef02815b5acfa1cacf9af7e1d0503fe49cc54393c1203169d818eb76876b
+size 402803885

model_state_layer_2_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b17079c4dd7c1adbc6b5cf907a51241d0f32d5cc95adae2f5a4d2b3e8a8e82e9
+size 402803874

model_state_layer_30_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a99c0b0b3bd02ce5b8026951b44b644e9451b938c310ba30b33c755ae92d0f63
+size 402803885

model_state_layer_31_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f82a2f6b819c97ee133d7fba1e761f761a6c2342330416da6cdc1847815ccbd7
+size 402803885

model_state_layer_32_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:605f545510dc2b8fa66c102da27fe2f15f3325e0452c3480e4b869874cc396c2
+size 402803885

model_state_layer_33_LayerNormWrapper.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0fef3ad9a684ad078dc8af6a7034775dfb3fb66550f2a44945928150a3c162ed
+size 9650

model_state_layer_34_TransformerLMHead.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:4001f5f525c856dbbc663e937ca635a73f20fa3469b8ba0701f6207ce50f7338
+size 536872360

model_state_layer_3_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dc16f965d119a504c06a73ddbafe4dad49203c9d3ef0d888f35148b96b6b9882
+size 402803874

model_state_layer_4_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1df0912a91b8a4eca662a6ee06c46f211b27269357ba4547ce5f506190a5444f
+size 402803874

model_state_layer_5_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:edeb1640ac3eebec2577490b14a5285d265d0c0a2940e2dd74c2b4ce828fcd5c
+size 402803874

model_state_layer_6_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d2180db549479bdf9deaa373c0f2844c1df3a5083e9e277e241871bda5794a82
+size 402803874

model_state_layer_7_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:12c8db34868983038b6a3b6189f2e8a7d875b59b86f06077cd765b97fb023ad6
+size 402803874

model_state_layer_8_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5655718812e3ae200b3cae538d2b20325fbe43f905bab66cfc7c7885e8ed20a9
+size 402803874

model_state_layer_9_TransformerLayer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:5a148acb4ef6d0054a3289dc80c99145e7f6459f2aeffc0f5daca91a2dbbb575
+size 402803874