From Babble to Words - a phonemetransformers Collection

updated Apr 3

The models, tokenizers and datasets used in From Babble to Words, one of the winning BabyLM 2024 submissions, exploring phoneme-based training.

Upvote

From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes

Paper • 2410.22906 • Published Oct 30, 2024
phonemetransformers/IPA-BabyLM

Viewer • Updated Apr 8 • 12.5M • 150 • 1
phonemetransformers/IPA-BabyLM-evaluation

Preview • Updated Apr 3 • 231
phonemetransformers/babble-tokenizers

Updated Apr 3
phonemetransformers/GPT2-85M-BPE-PHON

0.1B • Updated Apr 3 • 33

Note GPT2 with 85M non-embedding parameters trained using the BPE-PHON tokenizer.
phonemetransformers/GPT2-85M-BPE-PHON-SPACELESS

0.1B • Updated Apr 3 • 13

Note GPT2 with 85M non-embedding parameters trained using the BPE-PHON-SPACELESS tokenizer.
phonemetransformers/GPT2-85M-CHAR-TXT-SPACELESS

0.1B • Updated Apr 3 • 3

Note GPT2 with 85M non-embedding parameters trained using the CHAR-TXT-SPACELESS tokenizer.
phonemetransformers/GPT2-85M-CHAR-PHON

0.1B • Updated Apr 3 • 20

Note GPT2 with 85M non-embedding parameters trained using the CHAR-PHON tokenizer.
phonemetransformers/GPT2-85M-CHAR-PHON-SPACELESS

0.1B • Updated Apr 3 • 22

Note GPT2 with 85M non-embedding parameters trained using the CHAR-PHON-SPACELESS tokenizer.
phonemetransformers/GPT2-85M-CHAR-TXT

0.1B • Updated Apr 3 • 167

Note GPT2 with 85M non-embedding parameters trained using the CHAR-TXT tokenizer.
phonemetransformers/GPT2-85M-BPE-TXT-SPACELESS

0.1B • Updated Apr 3 • 4

Note GPT2 with 85M non-embedding parameters trained using the BPE-TXT-SPACELESS tokenizer.
phonemetransformers/GPT2-85M-BPE-TXT

0.1B • Updated Apr 3 • 5.37k • 1

Note GPT2 with 85M non-embedding parameters trained using the BPE-TXT tokenizer.

Upvote