Hugging Face's logo

SmolEvalData

non-profit
Activity Feed

AI & ML interests

None defined yet.

Recent Activity

nouamanetaziΒ  authored a paper about 2 months ago
SmolVLM: Redefining small and efficient multimodal models
anton-lΒ  authored a paper about 2 months ago
SmolVLM: Redefining small and efficient multimodal models
eliebakΒ  authored a paper about 2 months ago
SmolVLM: Redefining small and efficient multimodal models
View all activity

SmolEvalData's activity

loubnabnlΒ 
posted an update 8 days ago
nouamanetaziΒ 
authored a paper about 2 months ago
anton-lΒ 
authored a paper about 2 months ago
eliebakΒ 
authored a paper about 2 months ago
loubnabnlΒ 
authored a paper about 2 months ago
eliebakΒ 
posted an update 2 months ago
view post
Post
1771
Google just dropped an exciting technical report for the brand-new Gemma3 model! πŸš€ Here are my personal notes highlighting the most intriguing architectural innovations, design choices, and insights from this release:

1) Architecture choices:
> No more softcaping, replace by QK-Norm
> Both Pre AND Post Norm
> Wider MLP than Qwen2.5, ~ same depth
> SWA with 5:1 and 1024 (very small and cool ablation on the paper!)
> No MLA to save KV cache, SWA do the job!

2) Long context
> Only increase the rope in the global layer (to 1M)
> Confirmation that it's harder to do long context for smol models, no 128k for the 1B
> Pretrained with 32k context? seems very high
> No yarn nor llama3 like rope extension

3) Distillation
> Only keep te first 256 logits for the teacher
> Ablation on the teacher gap (tl;dr you need some "patience" to see that using a small teacher is better)
> On policy distillation yeahh (by
@agarwl_
et al), not sure if the teacher gap behave the same here, curious if someone have more info?

4) Others
> Checkpoint with QAT, that's very cool
> RL using improve version of BOND, WARM/WARP good excuse to look at
@ramealexandre
papers
> Only use Zero3, no TP/PP if i understand correctly ?
> Training budget relatively similar than gemma2
  • 1 reply
Β·
eliebakΒ 
authored a paper 4 months ago
loubnabnlΒ 
authored a paper 4 months ago
eliebakΒ 
authored a paper 4 months ago
anton-lΒ 
authored a paper 4 months ago
anton-lΒ 
posted an update 5 months ago
view post
Post
2879
Introducing πŸ“π…π’π§πžπŒπšπ­π‘: the best public math pre-training dataset with 50B+ tokens!
HuggingFaceTB/finemath

Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.

We build the dataset by:
πŸ› οΈ carefully extracting math data from Common Crawl;
πŸ”Ž iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction.

We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.

We hope this helps advance the performance of LLMs on math and reasoning! πŸš€
We’re also releasing all the ablation models as well as the evaluation code.

HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
loubnabnlΒ 
posted an update 6 months ago
view post
Post
3705
Making SmolLM2 reproducible: open-sourcing our training & evaluation toolkit πŸ› οΈ https://github.com/huggingface/smollm/

- Pre-training code with nanotron
- Evaluation suite with lighteval
- Synthetic data generation using distilabel (powers our new SFT dataset HuggingFaceTB/smoltalk)
- Post-training scripts with TRL & the alignment handbook
- On-device tools with llama.cpp for summarization, rewriting & agents

Apache 2.0 licensed. V2 pre-training data mix coming soon!

Which other tools should we add next?
anton-lΒ 
authored a paper 11 months ago
loubnabnlΒ 
authored a paper 11 months ago
eliebakΒ 
posted an update 11 months ago
view post
Post
1835
Wow, impressive 340B model by nvidia with a nice permissive license! πŸš€ The technical report is full of insights and seems to use a different learning rate schedule than cosine, probably a variant of WSD. Hope to get more info on that! πŸ‘€

nvidia/nemotron-4-340b-666b7ebaf1b3867caf2f1911
loubnabnlΒ 
posted an update 12 months ago
view post
Post
5890
🍷 FineWeb technical report is out and so is πŸ“š FineWeb-Edu, a 1.3 trillion tokens dataset that outperforms all other open web datasets, with remarkable improvements on educational benchmarksΒ such as MMLU, ARC, and OpenBookQA.

Technical report: HuggingFaceFW/blogpost-fineweb-v1
Dataset: HuggingFaceFW/fineweb-edu

We used Llama 3 generations to train an educational quality classifier, filtering the 15 trillion tokens of FineWeb to select only those with high educational value (an approach also used in Llama 3 and Phi-3 training datasets). We're releasing both FineWeb-Edu and the classifier, along with a larger, less heavily filtered version containing 5.4 trillion tokens.

You can find more details about the dataset and the experiments we ran in the FineWeb technical report, It's a 45-minute read but it contains all the secret sauce for building high quality web datasets.

Enjoy!
eliebakΒ 
authored a paper 12 months ago
loubnabnlΒ 
authored a paper 12 months ago
stillermanΒ 
authored a paper about 1 year ago
loubnabnlΒ 
posted an update about 1 year ago
view post
Post
6444
We've just published a detailed blog post on the creation of Cosmopedia dataset. We hope this will provide insights about generating synthetic data at scale for pre-training.
https://huggingface.co/blog/cosmopedia

Here are some key takeaways:
🎯 Prompt curation is crucial: we want to cover many topics with few duplicates.
πŸ“š You can leverage various resources for diversity: using different seed data, generation formats, and target audiences.
βš™οΈ The importance of a good technical stack: for scalable generations with tools like llm-swarm and fast model training and evaluation.

Have a good read!
  • 1 reply
Β·