Spaces:
Running
Running
readme: emojify more headers π
Browse files
README.md
CHANGED
@@ -17,7 +17,7 @@ The following LMs are currently supported:
|
|
17 |
* [Token Dropping for efficient BERT Pretraining](https://aclanthology.org/2022.acl-long.262/) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/token-dropping-bert)
|
18 |
* [Training ELECTRA Augmented with Multi-word Selection](https://aclanthology.org/2021.findings-acl.219/) (TEAMS) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/teams)
|
19 |
|
20 |
-
# FineWeb-LMs
|
21 |
|
22 |
Following LMs were pretrained on the (10BT subset) of the famous [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) and [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset:
|
23 |
|
@@ -25,7 +25,7 @@ Following LMs were pretrained on the (10BT subset) of the famous [FineWeb](https
|
|
25 |
* Token Dropping BERT-based - find the [best model checkpoint here](https://huggingface.co/model-garden-lms/bert-base-token-dropping-finewebs-901k)
|
26 |
* TEAMS-based - fine the [best model checkpoint here](https://huggingface.co/model-garden-lms/teams-base-finewebs-1m)
|
27 |
|
28 |
-
# ScandEval Evaluation
|
29 |
|
30 |
To find the best checkpoints and compare our FineWeb-LMs to other models (BERT, ELECTRA and RoBERTa) we perform an evaluation using the great [ScandEval](https://github.com/ScandEval/ScandEval) library.
|
31 |
|
|
|
17 |
* [Token Dropping for efficient BERT Pretraining](https://aclanthology.org/2022.acl-long.262/) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/token-dropping-bert)
|
18 |
* [Training ELECTRA Augmented with Multi-word Selection](https://aclanthology.org/2021.findings-acl.219/) (TEAMS) - see [pretraining instructions](https://github.com/stefan-it/model-garden-lms/tree/main/teams)
|
19 |
|
20 |
+
# π· FineWeb-LMs
|
21 |
|
22 |
Following LMs were pretrained on the (10BT subset) of the famous [FineWeb](https://huggingface.co/datasets/HuggingFaceFW/fineweb) and [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu) dataset:
|
23 |
|
|
|
25 |
* Token Dropping BERT-based - find the [best model checkpoint here](https://huggingface.co/model-garden-lms/bert-base-token-dropping-finewebs-901k)
|
26 |
* TEAMS-based - fine the [best model checkpoint here](https://huggingface.co/model-garden-lms/teams-base-finewebs-1m)
|
27 |
|
28 |
+
# π ScandEval Evaluation
|
29 |
|
30 |
To find the best checkpoints and compare our FineWeb-LMs to other models (BERT, ELECTRA and RoBERTa) we perform an evaluation using the great [ScandEval](https://github.com/ScandEval/ScandEval) library.
|
31 |
|