stefan-it commited on
Commit
613427f
Β·
verified Β·
1 Parent(s): b4b321f

readme: emojify more headers πŸ’•

Browse files
Files changed (1) hide show
  1. README.md +2 -2
README.md CHANGED
@@ -17,7 +17,7 @@ The following LMs are currently supported:
17
  * [Token Dropping for efficient BERT Pretraining](https://aclanthology.org/2022.acl-long.262/) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/token-dropping-bert)
18
  * [Training ELECTRA Augmented with Multi-word Selection](https://aclanthology.org/2021.findings-acl.219/) (TEAMS) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/teams)
19
 
20
- # FineWeb-LMs
21
 
22
  Following LMs were pretrained on the (10BT subset) of the famous [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) and [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset:
23
 
@@ -25,7 +25,7 @@ Following LMs were pretrained on the (10BT subset) of the famous [FineWeb](https
25
  * Token Dropping BERT-based - find the [best model checkpoint here](https://huggingface.co/model-garden-lms/bert-base-token-dropping-finewebs-901k)
26
  * TEAMS-based - fine the [best model checkpoint here](https://huggingface.co/model-garden-lms/teams-base-finewebs-1m)
27
 
28
- # ScandEval Evaluation
29
 
30
  To find the best checkpoints and compare our FineWeb-LMs to other models (BERT, ELECTRA and RoBERTa) we perform an evaluation using the great [ScandEval](https://github.com/ScandEval/ScandEval) library.
31
 
 
17
  * [Token Dropping for efficient BERT Pretraining](https://aclanthology.org/2022.acl-long.262/) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/token-dropping-bert)
18
  * [Training ELECTRA Augmented with Multi-word Selection](https://aclanthology.org/2021.findings-acl.219/) (TEAMS) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/teams)
19
 
20
+ # 🍷 FineWeb-LMs
21
 
22
  Following LMs were pretrained on the (10BT subset) of the famous [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) and [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset:
23
 
 
25
  * Token Dropping BERT-based - find the [best model checkpoint here](https://huggingface.co/model-garden-lms/bert-base-token-dropping-finewebs-901k)
26
  * TEAMS-based - fine the [best model checkpoint here](https://huggingface.co/model-garden-lms/teams-base-finewebs-1m)
27
 
28
+ # πŸ“Š ScandEval Evaluation
29
 
30
  To find the best checkpoints and compare our FineWeb-LMs to other models (BERT, ELECTRA and RoBERTa) we perform an evaluation using the great [ScandEval](https://github.com/ScandEval/ScandEval) library.
31