File size: 420 Bytes
8e1c2bb 56cc2c6 8e1c2bb 19df664 5fda600 19df664 8e1c2bb 19df664 5872d14 8e1c2bb 19df664 8e1c2bb |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
---
license: mit
datasets:
- allenai/c4
language:
- en
library_name: transformers
---
# Bingus-v0.1-60M-Base
A not-so-state-of-the-art 60M parameter transformer model.
Uses the olmo default architecture.
### Specs
Heads: 8
Layers: 8
Dimension model: 512
Dimension mlp: 4096
eval/v3-small-c4_en-validation/Perplexity: 40.33
### Training Data
Pretraining:
- 5B Tokens C4 (preprocessed, from olmo-data.org) |