Pre-trained models in MiniPLM: Knowledge Distillation for Pre-Training Language Models
AI & ML interests
Training efficient language models (MiniLLM, MiniPLM)
Organization Card
models
50

MiniLLM/MiniLLM-gpt2-340M
Text Generation
•
Updated
•
30
•
4

MiniLLM/SFT-gpt2-120M
Text Generation
•
0.1B
•
Updated
•
11

MiniLLM/SFT-gpt2-760M
Text Generation
•
0.8B
•
Updated
•
26

MiniLLM/MiniPLM-Qwen-500M
Text Generation
•
0.5B
•
Updated
•
24
•
7

MiniLLM/MiniPLM-llama3.1-212M
Text Generation
•
0.2B
•
Updated
•
9
•
4

MiniLLM/MiniPLM-Mamba-130M
Text Generation
•
0.1B
•
Updated
•
17
•
3

MiniLLM/MiniPLM-Qwen-1.2B
Text Generation
•
1B
•
Updated
•
9
•
3

MiniLLM/Ref-Pretrain-Qwen-104M
Text Generation
•
0.1B
•
Updated
•
59
•
2

MiniLLM/Pretrain-Qwen-1.2B
Text Generation
•
1B
•
Updated
•
9

MiniLLM/Pretrain-Qwen-500M
Text Generation
•
0.5B
•
Updated
•
8
datasets
10
MiniLLM/pile-diff_samp-qwen_1.8B-qwen_104M-r0.5
Updated
•
105
MiniLLM/pile-tokenized
Updated
•
78
•
2
MiniLLM/roberta-corpus-processed
Updated
•
20
MiniLLM/openwebtext-processed
Updated
•
33
MiniLLM/dolly-processed
Viewer
•
Updated
•
110k
•
144
•
1
MiniLLM/sinst
Viewer
•
Updated
•
8.35k
•
52
•
1
MiniLLM/uinst
Viewer
•
Updated
•
64.8k
•
58
•
1
MiniLLM/self-inst
Viewer
•
Updated
•
242
•
38
•
2
MiniLLM/Vicuna
Viewer
•
Updated
•
80
•
43
•
1
MiniLLM/dolly
Viewer
•
Updated
•
500
•
83