Should We Still Pretrain Encoders with Masked Language Modeling?
Nicolas-BZRD
Nicolas-BZRD
AI & ML interests
PhD Student | NLP - LLMs - Adaptation real-world problem
Optimization
Recent Activity
upvoted
an
article
5 days ago
Turk-LettuceDetect: A Hallucination Detection Models for Turkish RAG Applications
liked
a dataset
21 days ago
nvidia/Llama-Nemotron-VLM-Dataset-v1
Organizations
The ULD loss, based on optimal transport, enables distillation across different LLM families without requiring shared tokenizers.
-
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
Paper • 2402.12030 • Published • 1 -
mistralai/Mistral-7B-Instruct-v0.2
Text Generation • 7B • Updated • 462k • • 2.95k -
meta-llama/Llama-2-7b-chat-hf
Text Generation • 7B • Updated • 1.01M • 4.57k -
EleutherAI/pythia-160m-deduped
Text Generation • 0.2B • Updated • 147k • 3
Scaling Multilingual Encoders for European Languages.
CroissantLLM: A Truly Bilingual French-English Language Model
-
CroissantLLM: A Truly Bilingual French-English Language Model
Paper • 2402.00786 • Published • 27 -
croissantllm/CroissantLLMChat-v0.1
Text Generation • 1B • Updated • 3.15k • 51 -
croissantllm/CroissantLLMBase
Text Generation • Updated • 1.53k • 33 -
croissantllm/croissant_dataset
Viewer • Updated • 16.7B • 1.19k • 7
Should We Still Pretrain Encoders with Masked Language Modeling?
Scaling Multilingual Encoders for European Languages.
The ULD loss, based on optimal transport, enables distillation across different LLM families without requiring shared tokenizers.
-
Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
Paper • 2402.12030 • Published • 1 -
mistralai/Mistral-7B-Instruct-v0.2
Text Generation • 7B • Updated • 462k • • 2.95k -
meta-llama/Llama-2-7b-chat-hf
Text Generation • 7B • Updated • 1.01M • 4.57k -
EleutherAI/pythia-160m-deduped
Text Generation • 0.2B • Updated • 147k • 3
CroissantLLM: A Truly Bilingual French-English Language Model
-
CroissantLLM: A Truly Bilingual French-English Language Model
Paper • 2402.00786 • Published • 27 -
croissantllm/CroissantLLMChat-v0.1
Text Generation • 1B • Updated • 3.15k • 51 -
croissantllm/CroissantLLMBase
Text Generation • Updated • 1.53k • 33 -
croissantllm/croissant_dataset
Viewer • Updated • 16.7B • 1.19k • 7