Towards Best Practices for Open Datasets for LLM Training
Paper
β’
2501.08365
β’
Published
β’
40
StarCoder2 Data
community
AI & ML interests
None defined yet.
Recent Activity
View all activity
starcoder2data's activity
hynkyΒ
authored
a
paper
2 days ago
lvwerraΒ
authored
a
paper
2 days ago
Post
2235
Introducing ππ
π’π§πππππ‘: the best public math pre-training dataset with 50B+ tokens!
HuggingFaceTB/finemath
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
We build the dataset by:
π οΈ carefully extracting math data from Common Crawl;
π iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction.
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
We hope this helps advance the performance of LLMs on math and reasoning! π
Weβre also releasing all the ablation models as well as the evaluation code.
HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
HuggingFaceTB/finemath
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
We build the dataset by:
π οΈ carefully extracting math data from Common Crawl;
π iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction.
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
We hope this helps advance the performance of LLMs on math and reasoning! π
Weβre also releasing all the ablation models as well as the evaluation code.
HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
Post
1905
Making SmolLM2 reproducible: open-sourcing our training & evaluation toolkit π οΈ https://github.com/huggingface/smollm/
- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents
Apache 2.0 licensed. V2 pre-training data mix coming soon!
Which other tools should we add next?
- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents
Apache 2.0 licensed. V2 pre-training data mix coming soon!
Which other tools should we add next?
lvwerraΒ
authored
a
paper
3 months ago
lvwerraΒ
authored
a
paper
7 months ago
anton-lΒ
authored
a
paper
7 months ago
hynkyΒ
authored
a
paper
7 months ago
loubnabnlΒ
authored
a
paper
7 months ago
joaomonteirofΒ
authored
a
paper
7 months ago
lvwerraΒ
updated
a
Space
8 months ago
Post
5574
π· FineWeb technical report is out and so is π FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarksΒ such as MMLU, ARC, and OpenBookQA.
Technical report: HuggingFaceFW/blogpost-fineweb-v1
Dataset: HuggingFaceFW/fineweb-edu
We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens.
You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
Enjoy!
Technical report: HuggingFaceFW/blogpost-fineweb-v1
Dataset: HuggingFaceFW/fineweb-edu
We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens.
You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.
Enjoy!
lvwerraΒ
authored
a
paper
8 months ago
loubnabnlΒ
authored
a
paper
8 months ago
joaomonteirofΒ
authored
a
paper
9 months ago
Post
6405
We've just published a detailed blog post on the creation of Cosmopedia dataset. We hope this will provide insights about generating synthetic data at scale for pre-training.
https://huggingface.co/blog/cosmopedia
Here are some key takeaways:
π― Prompt curation is crucial: we want to cover many topics with few duplicates.
π You can leverage various resources for diversity: using different seed data, generation formats, and target audiences.
βοΈ The importance of a good technical stack: for scalable generations with tools like llm-swarm and fast model training and evaluation.
Have a good read!
https://huggingface.co/blog/cosmopedia
Here are some key takeaways:
π― Prompt curation is crucial: we want to cover many topics with few duplicates.
π You can leverage various resources for diversity: using different seed data, generation formats, and target audiences.
βοΈ The importance of a good technical stack: for scalable generations with tools like llm-swarm and fast model training and evaluation.
Have a good read!
- 1 reply
anton-lΒ
authored
a
paper
11 months ago
loubnabnlΒ
authored
a
paper
11 months ago
lvwerraΒ
authored
a
paper
11 months ago
Post
β Today weβre releasing The Stack v2 & StarCoder2: a series of 3B, 7B & 15B code generation models trained on 3.3 to 4.5 trillion tokens of code:
- StarCoder2-15B matches or outperforms CodeLlama 34B, and approaches DeepSeek-33B on multiple benchmarks.
- StarCoder2-3B outperforms StarCoderBase-15B and similar sized models.
- The Stack v2 a 4x larger dataset than the Stack v1, resulting in 900B unique code tokens π
As always, we released everything from models and datasets to curation code. Enjoy!
π StarCoder2 collection: bigcode/starcoder2-65de6da6e87db3383572be1a
π Paper: https://drive.google.com/file/d/17iGn3c-sYNiLyRSY-A85QOzgzGnGiVI3/view
π BlogPost: https://huggingface.co/blog/starcoder2
π Code Leaderboard: bigcode/bigcode-models-leaderboard
- StarCoder2-15B matches or outperforms CodeLlama 34B, and approaches DeepSeek-33B on multiple benchmarks.
- StarCoder2-3B outperforms StarCoderBase-15B and similar sized models.
- The Stack v2 a 4x larger dataset than the Stack v1, resulting in 900B unique code tokens π
As always, we released everything from models and datasets to curation code. Enjoy!
π StarCoder2 collection: bigcode/starcoder2-65de6da6e87db3383572be1a
π Paper: https://drive.google.com/file/d/17iGn3c-sYNiLyRSY-A85QOzgzGnGiVI3/view
π BlogPost: https://huggingface.co/blog/starcoder2
π Code Leaderboard: bigcode/bigcode-models-leaderboard